Feed aggregator

Ubuntu App Developer Blog: A Fast Thumbnailer for Ubuntu

Planet Ubuntu - Sun, 08/16/2015 - 23:45

Over the past few months, James Henstridge, Xavi Garcia Mena, and I have implemented a fast and scalable thumbnailing service for Ubuntu and Ubuntu Touch. This post explains how we did it, and how we achieved our performance and reliability goals.

Introduction

On a phone as well as the desktop, applications need to display image thumbnails for various media, such as photos, songs, and videos. Creating thumbnails for such media is CPU-intensive and can be costly in bandwidth if images are retrieved over the network. In addition, different types of media require the use of different APIs that are non-trivial to learn. It makes sense to provide thumbnail creation as a platform API that hides this complexity from application developers and, to improve performance, to cache thumbnails on disk.

This article explains the requirements we had and how we implemented a thumbnailer service that is extremely fast and scalable, and robust in the face of power loss or crashes.

Requirements

We had a number of requirements we wanted to meet in our implementation.

  • Robustness
    In the event of a crash, the implementation must guarantee the integrity of on-disk data structures. This is particularly important on a phone, where we cannot expect the user to perform manual recovery (such as cleaning up damaged files). Because batteries can run out at any time, integrity must be guaranteed even in the face of power loss.
  • Scalability
    It is common for people to store many thousands of songs and photos on a device, so the cache must scale to at least tens of thousands of records. Thumbnails can range in size from a few kilobytes to well over a megabyte (for “thumbnails” at full-screen resolution), so the cache must deal efficiently with large records.
  • Re-usability
    Persistent and reliable on-disk storage of arbitrary records (ranging in size from a few bytes to potentially megabytes) is a common application requirement, so we did not want to create a cache implementation that is specific to thumbnails. Instead, the disk cache is provided as a stand-alone C++ API that can be used for any number of other purposes, such as a browser or HTTP cache, or to build an object file cache similar to ccache.
  • High performance
    The performance of the thumbnailer directly affects the user experience: it is not nice for the customer to look at “please wait a while” icons in, say, an image gallery while thumbnails are being loaded one by one. We therefore had to have a high-performance implementation that delivers cached thumbnails quickly (on the order of a millisecond per thumbnail on an Arm CPU). An efficient implementation also helps to conserve battery life.
  • Location independence and extensibility
    Canonical runs an image server at dash.ubuntu.com that provides album and artist artwork for many musicians and bands. Images from this server are used to display artwork in the music player for media that contains ID3 tags, but does not embed artwork in the media file. The thumbnailer must work with embedded images as well as remote images, and it must be possible to extend it for new types of media without unduly disturbing the existing code.
  • Low bandwidth consumption
    Mobile phones typically come with data caps, so the cache has to be frugal with network bandwidth.
  • Concurrency and isolation
    The implementation has to allow concurrent access by multiple applications, as well as concurrent access from a single implementation. Besides needing to be thread-safe, this means that a request for a thumbnail that is slow (such as downloading an image over the network) must not delay other requests.
  • Fault tolerance
    Mobile devices lose network access without warning, and users can add corrupt media files to their device. The implementation must be resilient to partial failures, such as incomplete network replies, dropped connections, and bad image data. Moreover, the recovery strategy for such failures must conserve battery and avoid repeated futile attempts to create thumbnails from media that cannot be retrieved or contains malformed data.
  • Security
    The implementation must ensure that applications cannot see (or, worse, overwrite) each other’s thumbnails or coerce the thumbnailer into delivering images from files that an application is not allowed to read.
  • Asynchronous API
    The customers of the thumbnailer are applications that are written in QML or Qt, which cannot block in the UI thread. The thumbnailer therefore must provide a non-blocking API. Moreover, the application developer should be able to get the best possible performance without having to use threads. Instead, concurrency must be internal to the implementation (which is able to put threads to use intelligently where they make sense), instead of the application throwing threads at the problem in the hope that it might make things faster when, in fact, it might just add overhead.
  • Monitoring
    The effectiveness of a cache cannot be assessed without statistics to show hit and miss rates, evictions, and other basic performance data, so it must provide a way to extract this information.
  • Error reporting
    When something goes wrong with a system service, typically the only way to learn about the problem is to look at log messages. In case of a failure, the implementation must leave enough footprints behind to allow someone to diagnose a failure after the fact with some chance of success.
  • Backward compatibility
    This project was a rewrite of an earlier implementation. Rather than delivering a “big bang” piece of software and potentially upsetting existing clients, we incrementally changed the implementation such that existing applications continued to work. (The only pre-existing interface was a QML interface that required no change.)
System architecture

Here is a high-level overview of the main system components.

External API

To the outside world, the thumbnailer provides two APIs.

One API is a QML plugin that registers itself as an image provider for QQuickAsyncImageProvider. This allows the caller to to pass a URI that encodes a query for a local or remote thumbnail at a particular size; if the URI matches the registered provider, QML transfers control to the entry points in our plugin.

The second API is a Qt API that provides three methods:

QSharedPointer<Request> getThumbnail(QString const& filePath, QSize const& requestedSize); QSharedPointer<Request> getAlbumArt(QString const& artist, QString const& album, QSize const& requestedSize); QSharedPointer<Request> getArtistArt(QString const& artist, QString const& album, QSize const& requestedSize);

The getThumbnail() method extracts thumbnails from local media files, whereas getAlbumArt() and getArtistArt() retrieve artwork from the remote image server. The returned Request object provides a finished signal, and methods to test for success or failure of the request and to extract a thumbnail as a QImage. The request also provides a waitForFinished() method, so the API can be used synchronously.

Thumbnails are delivered to the caller in the size they are requested, subject to a (configurable) 1920-pixel limit. As an escape hatch, requests with width and height of zero deliver artwork at its original size, even if it exceeds the 1920-pixel limit. The scaling algorithm preserves the original aspect ratio and never scales up from the original, so the returned thumbnails may be smaller than their requested size.

DBus service

The thumbnailer is implemented as a DBus service with two interfaces. The first interface provides the server-side implementation of the three methods of the external API; the second interface is an administrative interface that can deliver statistics, clear the internal disk caches, and shut down the service. A simple tool, thumbnailer-admin, allows both interfaces to be called from the command line.

To conserve resources, the service is started on demand by DBus and shuts down after 30 seconds of idle time.

Image extraction

Image extraction uses an abstract base class. This interface is independent of media location and type. The actual image extraction is performed by derived implementations that download images from the remote server, extract them from local image files, or extract them from local streaming media files. This keeps knowledge of image location and encoding out of the main caching and error handling logic, and allows us to support new media types (whether local or remote) by simply adding extra derived implementations.

Image extraction is asynchronous, with currently three implementations:

  • Image downloader
    To retrieve artwork from the remote image server, the service talks to an abstract base class with asynchronous download_album() and download_artist() methods. This allows multiple downloads to run concurrently and makes it easy to add new local or remote image providers without disturbing the code for existing ones. A class derived from that abstract base implements a REST API with QNetworkAccessManager to retrieve images from dash.ubuntu.com.
  • Photo extractor
    The photo extractor is responsible for delivering images from local image files, such as JPEG or PNG files. It simply delegates that work to the image converter and scaler.
  • Audio and video thumbnail extractor
    To extract thumbnails from audio and video files, we use GStreamer. Due to reliability problems with some codecs that can hang or crash, we delegate the task to a separate vs-thumb executable. This shields the service from failures and also allows us to run several GStreamer pipelines concurrently without a crash of one pipeline affecting the others.
Image converter and scaler

We use a simple Image class with a synchronous interface to convert and scale different image formats to JPEG. The implementation uses Gdk-Pixbuf, which can handle many different input formats and is very efficient.

For JPEG source images, the code checks for the presence of EXIF data using libexif and, if it contains a thumbnail that is at least as large as the requested size, scales the thumbnail from the EXIF data. (For images taken with the camera on a Nexus 4, the original image size is 3264×1836, with an embedded EXIF thumbnail of 512×288. Scaling from the EXIF thumbnail is around one hundred times faster than scaling from the full-size image.)

Disk cache

The thumbnailer service optimizes performance and conserves bandwidth and battery by adopting a layered caching strategy.

Two-level caching with failure lookup

Internally, the service uses three separate on-disk caches:

  • Full-size cache
    This cache stores images that are expensive to retrieve (images that are remote or are embedded in audio and video files) at original resolution (scaled down to a 1920-pixel bounding box if the original image is larger). The default size of this cache is 50 MB, which is sufficient to hold around 400 images at 1920×1080 resolution. Images are stored in JPEG format (at a 90% quality setting).
  • Thumbnail cache
    This cache stores thumbnails at the size that was requested by the caller, such as 512×288. The default size of this cache is 100 MB, which is sufficient to store around 11,000 thumbnails at 512×288, or around 25,000 thumbnails at 256×144.
  • Failure cache
    The failure cache stores the keys for images that could not be extracted because of a failure. For remote images, this means that the server returned an authoritative answer “no such image exists”, or that we encountered an unexpected (non-authoritative) failure, such as the server not responding or a DNS lookup timing out. For local images, it means either that the image data could not be processed because it is damaged, or that an audio file does not contain embedded artwork.

The full-size cache exists because it is likely that an application will request thumbnails at different sizes for the same image. For example, when scrolling through a list of songs that shows a small thumbnail of the album cover beside each song, the user is likely to select one of the songs to play, at which point the media player will display the same cover in a larger size. By keeping full-size images in a separate (smallish) cache, we avoid performing an expensive extraction or download a second time. Instead, we create additional thumbnails by scaling them from the full-size cache (which uses an LRU eviction policy).

The thumbnail cache stores thumbnails that were previously retrieved, also using LRU eviction. Thumbnails are stored as JPEG at the default quality setting of 75%, at the actual size that was requested by the caller. Storing JPEG images (rather than, say, PNG) saves space and increases cache effectiveness. (The minimal quality loss from compression is irrelevant for thumbnails). Because we store thumbnails at the size they are actually needed, we may have several thumbnails for the same image in the cache (each thumbnail at a different size). But applications typically ask for thumbnails in only a small number of sizes, and ask for different sizes for the same image only rarely. So, the slight increase in disk space is minor and amply repaid by applications not having to scale thumbnails after they receive them from the cache, which saves battery and achieves better performance overall.

Finally, the failure cache is used to stop futile attempts to repeatedly extract a thumbnail when we know that the attempt will fail. It uses LRU eviction with an expiry time for each entry.

Cache lookup algorithm

When asked for a thumbnail at a particular size, the lookup and thumbnail generation proceed as follows:

  1. Check if a thumbnail exists in the requested size in the thumbnail cache. If so, return it.
  2. Check if a full-size image for the thumbnail exists in the full-size cache. If so, scale the new thumbnail from the full-size image, add the thumbnail to the thumbnail cache, and return it.
  3. Check if there is an entry for the thumbnail in the failure cache. If so, return an error.
  4. Attempt to download or extract the original image for the thumbnail. If the attempt fails, add an entry to the failure cache and return an error.
  5. If the original image was delivered by the remote server or was extracted locally from streaming media, add it to the full-size cache.
  6. Scale the thumbnail to the desired size, add it to the thumbnail cache, and return it.

Note that these steps represent only the logical flow of control for a particular thumbnail. The implementation executes these steps concurrently for different thumbnails.

Designing for performance

Apart from fast on-disk caches (see below), the thumbnailer must make efficient use of I/O bandwidth and threads. This not only means making things fast, but also to not unnecessarily waste resources such as threads, memory, network connections, or file descriptors. Provided that enough requests are made to keep the service busy, we do not want it to ever wait for a download or image extraction to complete while there is something else that could be done in the mean time, and we want it to keep all CPU cores busy. In addition, requests that are slow (because they require a download or a CPU-intensive image extraction) must not block requests that are queued up behind them if those requests would result in cache hits that could be returned immediately.

To achieve a high degree of concurrency without blocking on long-running operations while holding precious resources, the thumbnailer uses a three-phase lookup algorithm:

  1. In phase 1, we look at the caches to determine if we have a hit or an authoritative miss. Phase 1 is very fast. (It takes around a millisecond to return a thumbnail from the cache on a Nexus 4.) However, cache lookup can briefly stall on disk I/O or require a lot of CPU to extract and scale an image. To get good performance, phase 1 requests are passed to a thread pool with as many threads as there are CPU cores. This allows the maximum number of lookups to proceed concurrently.
  2. Phase 2 is initiated if phase 1 determines that a thumbnail requires download or extraction, either of which can take on the order of seconds. (In case of extraction from local media, the task is CPU intensive; in case of download, most of the time is spent waiting for the reply from the server.) This phase is scheduled asynchronously from an event loop. This minimizes task switching and allows large numbers of requests to be queued while only using a few bytes for each request that is waiting in the queue.
  3. Phase 3 is really a repeat of phase 1: if phase 2 produces a thumbnail, it adds it to the cache; if phase 2 does not produce a thumbnail, it creates an entry in the failure cache. By simply repeating phase 1, the lookup then results in either a thumbnail or an error.

If phase 2 determines that a download or extraction is required, that work is performed concurrently: the service schedules several downloads and extractions in parallel. By default, it will run up to two concurrent downloads, and as many concurrent GStreamer pipelines as there are CPUs. This ensures that we use all of the available CPU cores. Moreover, download and extraction run concurrently with lookups for phase 1 and 3. This means that, even if a cache lookup briefly stalls on I/O, there is a good chance that another thread can make use of the CPU.

Because slow operations do not block lookup, this also ensures that a slow request does not stall requests for thumbnails that are already in the cache. In other words, it does not matter how many slow requests are in progress: requests that can be completed quickly are indeed completed quickly, regardless of what is going on elsewhere.

Overall, this strategy works very well. For example, with sufficient workload, the service achieves around 750% CPU utilization on an 8-core desktop machine, while still delivering cache hits almost instantaneously. (On a Nexus 4, cache hits take a little over 1 ms while concurrent extractions or downloads are in progress.)

A re-usable persistent cache for C++

The three internal caches are implemented by a small and flexible C++ API. This API is available as a separate reusable PersistentStringCache component (see persistent-cache-cpp) that provides a persistent store of arbitrary key–value pairs. Keys and values can be binary, and entries can be large. (Megabyte-sized values do not present a problem.)

The implementation uses leveldb, which provides a very fast NoSQL database that scales to multi-gigabyte sizes and provides integrity guarantees. In particular, if the calling process crashes, all inserts that completed at the API level will be intact after a restart. (In case of a power failure or kernel crash, a few buffered inserts can be lost, but the integrity of the database is still guaranteed.)

To use a cache, the caller instantiates it with a path name, a maximum size, and an eviction policy. The eviction policy can be set to either strict LRU (least-recently-used) or LRU with an expiry time. Once a cache reaches its maximum size, expired entries (if any) are evicted first and, if that does not free enough space for a new entry, entries are discarded in least-recently-used order until enough room is available to insert a new record. (In all other respects, expired entries behave like entries that were never added.)

A simple get/put API allows records to be retrieved and added, for example:

auto c = core::PersistentStringCache::open( “my_cache”, 100 * 1024 * 1024, core::CacheDiscardPolicy::lru_only); // Look for an entry and add it if there is a cache miss. string key = "Bjarne"; auto value = c->get(key); if (value) {    cout << key << ″: ″ << *value << endl; } else {    value = "C++ inventor";  // Provide a value for the key.    c->put(key, *value);     // Insert it. }

Running this program prints nothing on the first run, and “Bjarne: C++ inventor” on all subsequent runs.

The API also allows application-specific metadata to be added to records, provides detailed statistics, supports dynamic resizing of caches, and offers a simple adapter template that makes it easy to store complex user-defined types without the need to clutter the code with explicit serialization and deserialization calls. (In a pinch, if iteration is not needed, the cache can be used as a persistent map by setting an impossibly large cache size, in which case no records are ever evicted.)

Performance

Our benchmarks indicate good performance. (Figures are for an Intel Ivy Bridge i7-3770k 3.5 GHz machine with a 256 GB SSD.) Our test uses 60-byte string keys. Values are binary blobs filled with random data (so they are not compressible), 20 kB in size with a standard deviation of 7,000, so the majority of values are 13–27 kB in size. The cache size is 100 MB, so it contains around 5,000 records.

Filling the cache with 100 MB of records takes around 2.8 seconds. Thereafter, the benchmark does a random lookup with an 80% hit probability. In case of a cache miss, it inserts a new random record, evicting old records in LRU order to make room for the new one. For 100,000 iterations, the cache returns around 4,800 “thumbnails” per second, with an aggregate read/write throughput of around 93 MB/sec. At 90% hit rate, we see twice the performance at around 7,100 records/sec. (Writes are expensive once the cache is full due to the need to evict entries, which requires updating the main cache table as well as an index.)

Repeating the test with a 1 GB cache produces identical timings so (within limits) performance remains constant for large databases.

Overall, performance is restricted largely by the bandwidth to disk. With a 7,200 rpm disk, we measured around one third of the performance with an SSD.

Recovering from errors

The overall design of the thumbnailer delivers good performance when things work. However, our implementation has to deal with the unexpected, such as network requests that do not return responses, GStreamer pipelines that crash, request overload, and so on. What follows is a partial list of steps we took to ensure that things behave sensibly, particularly on a battery-powered device.

Retry strategy

The failure cache provides an effective way to stop the service from endlessly trying to create thumbnails that, in an earlier attempt, returned an error.

For remote images, we know that, if the server has (authoritatively) told us that it has no artwork for a particular artist or album, it is unlikely that artwork will appear any time soon. However, the server may be updated with more artwork periodically. To deal with this, we add an expiry time of one week to the entries in the failure cache. That way, we do not try to retrieve the same image again until at least one week has passed (and only if we receive a request for a thumbnail for that image again later).

As opposed to authoritative answers from the image server (“I do not have artwork for this artist.”), we can also encounter transient failures. For example, the server may currently be down, or there may be some other network-related issue. In this case, we remember the time of the failure and do not try to contact the remote server again for two hours. This conserves bandwidth and battery power.

The device may also disconnected from the network, in which case any attempt to retrieve a remote image is doomed. Our implementation returns failure immediately on a cache miss for a remote image if no network is present or the device is in flight mode. (We do not add an entry to the failure cache in this case).

For local files, we know that, if an attempt to get a thumbnail for a particular file has failed, future attempts will fail as well. This means that the only way for the problem to get fixed is by modifying or replacing the actual media file. To deal with this, we add the inode number, modification time, and inode modification time to the key for local images. If a user replaces, say, a music file with a new one that contains artwork, we automatically pick up the new version of the file because its key has changed; the old version will eventually fall out of the cache.

Download and extraction failures

We monitor downloads and extractions for timely completion. (Timeouts for downloads and extractions can be configured separately.) If the server does not respond within 10 seconds, we abandon the attempt and treat it it as a transient network error. Similarly, the vs-thumb processes that extract images from audio and video files can hang. We monitor these processes and kill them if they do not produce a result within 10 seconds.

Database corruption

Assuming an error-free implementation of leveldb, database corruption is impossible. However, in practice, an errant command could scribble over the database files. If leveldb detects that the database is corrupted, the recovery strategy is simple: we delete the on-disk cache and start again from scratch. Because the cache contents are ephemeral anyway, this is fine (other than slower operation until the working set of thumbnails makes it into the cache again).

Dealing with backlog

The asynchronous API provided by the service allows an application to submit an unlimited number of requests. Lots of requests happen if, for example, the user has inserted a flash card with thousands of photos into the device and then requests a gallery view for the collection. If the service’s client-side API blindly forwards requests via DBus, this causes a problem because DBus terminates the connection once there are more than around 400 outstanding requests.

To deal with this, we limit the number of outstanding requests to 200 and send another request via DBus only when an earlier request completes. Additional requests are queued in memory. Because this happens on the client side, the number of outstanding requests is limited only by the amount of memory that is available to the client.

A related problem arises if a client submits many requests for a thumbnail for the same image. This happens when, for example, the user looks at a list of tracks: tracks that belong to the same album have the same artwork. If artwork needs to be retrieved from the remote server, naively forwarding cache misses for each thumbnail to the server would end up re-downloading the same image several times.

We deal with this by maintaining an in-memory map of all remote download requests that are currently in progress. If phase 1 reports a cache miss, before initiating a download, we add the key for the remote image to the map and remove it again once the download completes. If more requests for the same image encounter a cache miss while the download for the original request is still in progress, the key for the in-progress download is still in the map, and we hold additional requests for the same image until the download completes. We then schedule the held requests as usual and create their thumbnails from the image that was cached by the first request.

Security

The thumbnailer runs with normal user privileges. We use AppArmor’s aa_query_label() function to verify that the calling client has read access to a file it wants a thumbnail for. This prevents one application from accessing thumbnails produced by a different application, unless both applications can read the original file. In addition, we place the entire service under an AppArmor profile to ensure that it can write only to its own cache directory.

Conclusion

Overall, we are very pleased with the overall design and performance of the thumbnailer. Each component has a clearly defined role with a clean interface, which made it easy for us to experiment and to refine the design as we went along. The design is extensible, so we can support additional media types or remote data sources without disturbing the existing code.

We used threads sparingly and only where we saw worthwhile concurrency opportunities. Using asynchronous interfaces for long-running operations kept resource usage to a minimum and allowed us to take advantage of I/O interleaving. In turn, this extracts the best possible performance from the hardware.

The thumbnailer now runs on Ubuntu Touch and is used by the gallery, camera, and music apps, as well as for all scopes that display media thumbnails.

This article has been originally published on Michi Henning's blog.

David Tomaschik: So, is Windows 10 Spying On You?

Planet Ubuntu - Sun, 08/16/2015 - 14:00

"Extraordinary claims require extraordinary evidence."

A few days ago, localghost.org posted a translation of a Czech article alledging Windows 10 "phones home" in a number of ways. I was a little surprised, and more than a little alarmed, by some of the claims. Rather than blindly repost the claims, I decided it would be a good idea to see what I could test for myself. Rob Seder has done similarly but I'm taking it a step further to look at the real traffic contents.

Tools & Setup

I'm running the Windows 10 Insider Preview (which, admittedly, may not be the same as the release, but it's what I had access to) in VirtualBox. The NIC on my Windows 10 VM is connected to an internal network to a Debian VM with my tools installed, which is in turn connected out to the internet. On the Debian VM, I'm using mitmproxy to perform a traffic MITM. I've also used VirtualBox's network tracing to collect additional data.

Currently, I have all privacy settings set to the default, but I am not signed into a Microsoft Live account. This is an attempt to replicate the findings from the original article. At the moment, I'm only looking at HTTP/HTTPS traffic in detail, even though the original article wasn't even specific enough to indicate what protocols were being used.

Claim 1. All text typed on the keyboard is sent to Microsoft

When typing into the search bar within the Start menu, an HTTPS request is sent after each character entered. Presumably this is to give web results along with local results, but the amount of additional metadata included is just mind-boggling. Here's what the request for such a search looks like (some headers modified):

GET /AS/API/WindowsCortanaPane/V2/Suggestions?qry=about&cp=5&cvid=ce8c2c3ad6704645bb207c0401d709aa&ig=7fdd08f6d6474ead86e3c71404e36dd6&cc=US&setlang=en-US HTTP/1.1 Accept: */* X-BM-ClientFeatures: FontV4, OemEnabled X-Search-SafeSearch: Moderate X-Device-MachineId: {73737373-9999-4444-9999-A8A8A8A8A8A8} X-BM-Market: US X-BM-DateFormat: M/d/yyyy X-Device-OSSKU: 48 X-Device-NetworkType: ethernet X-BM-DTZ: -420 X-BM-UserDisplayName: Tester X-DeviceID: 0100D33317836214 X-BM-DeviceScale: 100 X-Device-Manufacturer: innotek GmbH X-BM-Theme: ffffff;005a9e X-BM-DeviceDimensionsLogical: 320x622 X-BM-DeviceDimensions: 320x622 X-Device-Product: VirtualBox X-BM-CBT: 1439740000 X-Device-isOptin: false X-Device-Touch: false X-AIS-AuthToken: AISToken ApplicationId=25555555-ffff-4444-cccc-a7a7a7a7a7a7&ExpiresOn=1440301800&HMACSHA256=CS y7XaNyyCE8oAZPeN%2b6IJ4ZrpqDDRZUIJyKvrIKnTA%3d X-Device-ClientSession: 95290000000000000000000000000000 X-Search-AppId: Microsoft.Windows.Cortana_cw5n1h2txyewy!CortanaUI X-MSEdge-ExternalExpType: JointCoord X-MSEdge-ExternalExp: sup001,pleasenosrm40ct,d-thshld42,d-thshld77,d-thshld78 Referer: https://www.bing.com/ Accept-Language: en-US Accept-Encoding: gzip, deflate User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0; Cortana 1.4.8.152; 10.0.0.0.10240.21) like Gecko Host: www.bing.com Connection: Keep-Alive Cookie: SA_SUPERFRESH_SUPPRESS=SUPPRESS=0&LAST=1439745358300; SRCHD=AF=NOFORM; ...

In addition to my query, "about", it sends a "DeviceID", a "MachineId", the username I'm logged in as, the platform (VirtualBox), and a number of opaque identifiers in the query, the X-AIS-AuthToken, and the Cookies. That's a lot of information just to give you search results.

Claim 2. Telemetry including file metadata is sent to Microsoft

I searched for several movie titles, including "Mission Impossible", "Hackers", and "Inside Out." Other than the Cortana suggestions above, I didn't see any traffic pertaining to these searches. Certainly, I didn't see any evidence of uploading a list of multimedia files from my Windows 10 system, as described in the original post.

I also searched for a phone number in the edge browser, as described in the original post. (Specifically, I search for 867-5309.) The only related traffic I saw is traffic to the site on which I performed the search (yellowpages.com). No traffic containing that phone number went to any Microsoft-run server, as far as I can tell.

Claim 3. When a webcam is connected, 35MB of data gets sent

Nope. Not even close. I shut down the VM, started a new PCAP, restarted, and attached a webcam via USB forwarding in VirtualBox. After the drivers were fully installed, I shut down the system. The total size of the pcap was under 800k in size, a far cry from the claimed 35MB. Looking at mitmproxy and the pcap, the largest single connection was ~82kB in size. I have no idea what traffic he saw, but I saw no alarming connection related to plugging in a webcam. My best guess is maybe it's actually 35MB of download, and his webcam required a driver download. (Admittedly a large driver, but I've seen bigger.)

Claim 4. Everything said into a microphone is sent

Even when attempting to use the speech recognition in Windows, I saw nothing that was large enough to be audio spoken being transferred. Additionally, no intercepted HTTP or HTTPS traffic contained the raw words that I spoke to the voice recognition service. Maybe if signed in to Windows Live, Cortana performs uploads, but without being signed in, I saw nothing representative of the words I used with speech recognition.

Claim 5. Large volumes of data are uploaded when Windows is left unattended

I left Windows running for >1 hour while I went and had lunch. There were a small number of HTTP(s) requests, but they all seemed to be related to either updating the weather information displayed in the tiles or checking for new Windows updates. I don't know what the OP considers "large volumes", but I'm not seeing it.

Conclusion

The original post made some extraordinary claims, and I'm not seeing anything to the degree they claimed. To be sure, Windows 10 shares more data with Microsoft than I'd be comfortable with, particularly if Cortana is enabled, but it doesn't seem to be anything like the levels described in the article. I wish the original poster had posted more about the type of traffic he was seeing, the specific requests, or even his methodology for testing.

The only dubious behavior I observed was sending every keystroke in the Windows Start Menu to the servers, but I understand that combined Computer/Web search is being sold as a feature, and this is necessary for that feature. I don't know why all the metadata is needed, and it's possibly excessive, but this isn't the keylogger the original post claimed.

Unfortunately, it's impossible to disprove his claims, but if it's as bad as suggested, reproducing it should've been possible, and I've been unable to reproduce it. I encourage others to try it as well -- if enough of us do it, it should be possible to either confirm or strongly refute the original claims.

Benjamin Drung: DebConf 15

Planet Ubuntu - Sun, 08/16/2015 - 04:31

I am still alive and currently attending DebConf 15. Feel free to grab me for a talk. I am just shy, not antisocial.


Salih Emin: SHC: How to encrypt and convert a shell script into a binary executable

Planet Ubuntu - Sun, 08/16/2015 - 04:04

In this tutorial we will discuss how to protect the source code of a shell script by converting it to a C binary

Ronnie Tucker: Second Snappy Ubuntu Core 15.04 Stable Version Now Ready for Raspberry Pi 2

Planet Ubuntu - Sat, 08/15/2015 - 20:37

A new Snappy Ubuntu Core 15.04 has been announced for Raspberry Pi 2, and users have been advised to re-flash their systems as soon as possible.

Canonical arrived late with an OS for Raspberry Pi 2, but they didn’t just port the regular Ubuntu OS. They’ve published an image based on Snappy Ubuntu Core 15.04, their latest version of the Ubuntu OS, which is based on the new Snappy package and is mostly aimed at IoT devices.

IoT is a concept that brings together devices that can be connected online, whether it’s a mini-PC like Raspberry Pi 2 or a smart fridge. All of these devices can either run their own operating system or they can be governed by one. Snappy Ubuntu Core 15.04 comes with its own Software Center, with dedicated apps.

 

Source:
http://news.softpedia.com/news/second-snappy-ubuntu-core-15-04-stable-version-now-ready-for-raspberry-pi-2-488820.shtml

Submitted by: Silviu Stahie

David Tomaschik: Blue Team Player's Guide for Pros vs Joes CTF

Planet Ubuntu - Sat, 08/15/2015 - 12:15

I've played in Dichotomy's Pros v Joes CTF for the past 3 years -- which, I'm told, makes me the only player to have done so. It's an incredible CTF and dramatically different from any other that I've ever played. Dichotomy and I were having lunch at DEF CON when he said "You know what would be cool? A blue team player's guide." So, I give to you, the blue team player's guide to the Pros v Joes CTF.

Basics

First, a quick review of the basics: PvJ is a 2-day long CTF. On day one, blue teams operate in a strictly defensive role. Red cell will be coming after you relentlessly, and it's your job to keep them out. On day two, things take a turn for the red, with blue teams both responsible for defending their network, but also given the opportunity to attack the networks of the other blue teams. Offense is fun, but do keep in mind that you need some defense on day two. Your network will have been reset, so you'll need to re-harden all your systems!

Scoring is based on several factors. As of 2015, the first day score was based on flags (gain points for finding your own "integrity" flags, lose points for having flags stolen), service uptime (lose points for downtime), tickets (lose points for failing to complete required tasks in the environment), and beacons (lose points for red cell leaving "beacons" that phone home on your system, indicating ongoing compromise). Day two scoring is similar, but now you can earn points by stealing flags from other teams and placing your own beacons on their systems to phone home.

Make sure you read the rules when you play -- each year they're a little different, and things that have been done before may not fit within the rules -- or may not be to your advantage anymore!

The Environment

Before I start talking strategy, let's talk a little bit about the environment and what to expect. Of course, Dichotomy may have new tricks up his sleeve at any time, so you have to assume almost everything below will change.

Connecting to the environment requires connecting to an OpenVPN network that provides a route to your environment as well as all of the other teams -- think of it as a self-contained mini-internet. Within this environment is a large vCenter deployment, containing all of the blue team environments. You'll get access to the vCenter server, but only to your hosts of course.

Each team will have a /24 network to protect, containing a number of hosts. In 2015, there were a dozen or so hosts per network. All blue team networks contain the same systems, providing the same services, but they will have different flags and credentials. In front of your /24 is a Cisco ASA firewall. Yes, you get access to configure it, so it's good to have someone on your team who has seen one before. (My team found this out the hard way this year.)

While the exact hosts are likely to change each year, I feel pretty confident that many of the core systems are unlikely to be dramatically different. Some of the systems that have consistently been present include:

  • A Windows Server 2008 acting as a Domain Controller
  • Several Linux servers in multiple flavors: Ubuntu, CentOS, SuSE
  • A number of Windows XP machines
Responsibilties as a Blue Team Member

This is intended as a learning experience, so nobody expects you to show up knowing everything about attack and defense on every system. That's just not realistic. But you should show up prepared to play, and there's several things involved in being prepared:

  1. Make sure you introduce yourself to your team mates via the team mailing list. It's good to know who's who, what their skill sets are, and what they're hoping to get out of the CTF. (And yes, "have a good time" is a perfectly acceptable objective.)
  2. Have your machine set up in advance. Obviously, you're going to need a laptop to play in this CTF. It doesn't need to be the fastest or newest machine, but you should have a minimum toolkit:
    • OpenVPN to connect to the environment, configured and tested.
    • VMWare vSphere Client to connect to the vCenter host. (Windows only, so this might also call for a VM if your host OS is not Windows.)
    • An RDP Client of some sort is useful for the Windows machines in the environment.
    • Tools to map out your network, such as nmap, OpenVAS, or similar.
    • Attack tools for the 2nd day: Metasploit is always popular. If you're not familiar with metasploit, consider also installing Armitage, a popular GUI front end. I usually run Kali Linux on the bare metal and have a Windows VM just for vSphere client. Make sure you run OpenVPN on your host OS so that traffic from both the host and guest gets routed to the environment properly.
  3. Learn a few basics in advance. At a minimum, know how to connect both to Windows and Linux systems. Never used ssh before? Learn how in advance. There's a reading list at the bottom of this article with resources that will help you familiarize yourself with many of the aspects involved before the day of the event.
  4. You don't have to sit there at the table the entire day both days, but you should plan for the majority of the time. If you want to go see a talk, that works, but let somebody know and make sure you're not walking off with the only copy of a password.
Strategy

I could probably find a Sun Tzu quote that fits here (that is the security industry tradition, after all) but really, they're getting a bit over used. What is important is realizing that you're part of a team and that you'll either succeed as a team or fail. Whether you fail as a team or as a bunch of individuals with no coordination depends on a lot of things, but if you're not working together, you can be certain of failure.

With so many systems, there's a lot of work to be done. With Red Team on the advance, there's a lot of work to be done quickly. You need to make sure that quickly does not turn into chaotically. I suggest first identifying all of the basic hardening steps to be taken, then splitting the work among the team, making sure each task is "owned" by a team member. Some of the tasks might include:

  1. Configure the ASA firewall to only allow in SSH, RDP, and the services you need for scoring purposes. (Note that you're not allowed to add IP filtering against red cell, nor block beacons to the scoring server.)
  2. Change the passwords on each system (divide this up so each individual only has 1-2 systems to handle) and document them. (A Google spreadsheet works well here.)
  3. Start an nmap scan of your entire network as a baseline. (Doing this from one of the hosts within the environment will be much faster than doing it from your laptop over the VPN.)
  4. Start disabling unnecessary services. (Again, responsibility for 1-2 systems per team member.)

Remember: document what you do, especially the passwords!

For the second day, I recommend a roughly 80/20 split of your team, switching after the basic hardening is done. That is, for the first hour or so, 80% of your team should be hardening systems while 20% looks for the low-hanging fruit on other team's networks. This is a good opportunity to get an early foothold before they have the chance to harden. After the initial hardening (setup ASA, change passwords, etc.), you want to devote more resources to the offensive, but you still need some people looking after the home front.

Good communication is key throughout, but watch out how you handle it: you never know who's listening. One year we had an IRC channel where everyone connected (over SSL of course!) for coordination so we would leak less information.

Pro Tips

Some tips just didn't fit well into other parts of the guide, so I've compiled them here.

  • Changing all the passwords to one "standard" may be attractive, but it'll only take one keylogger from red cell to make you regret that decision.
  • Consider disabling password-based auth on the linux machines entirely, and use SSH keys instead.
  • The scoring bot uses usernames and passwords to log in to some services. Changing those passwords may have an adverse effect on your scoring. Find other ways to lock down those accounts.
  • Rotate roles, giving everyone a chance to go on both offense and defense.
Recommended Reading Hardening Offensive Security Conclusion

The most important thing to remember is that you're there to learn and have fun. It doesn't matter if you win or lose, so long as you got something out of it. Three years straight, I've walked away from the table feeling like I got something out of it. I've met great people, had great fun, and learned a few things along the way. GL;HF!

Nathan Haines: Reintroducing the Ubuntu Free Culture Showcase

Planet Ubuntu - Sat, 08/15/2015 - 03:19

In the past, we’ve had the opportunity to showcase some really fun, really incredible media in Ubuntu. Content creators who value free culture have offered beautiful photography for the desktop and entertaining videos and music.

Not only does this highlight the fantastic work that comes out of free culture on Ubuntu desktops worldwide, but the music and video selections also help show off Ubuntu’s fantastic multimedia support by providing content for the Ubuntu live images.

The wallpaper contest has continued from cycle to cycle, but the audio and video contests have fallen by the wayside. But Ubuntu is more popular than ever, and can now power phones and tablets as well as desktops and laptops. So as we move closer towards a goal of convergence, we’d like to bring back this opportunity for artists to share their work with millions of Ubuntu users around the world.

All content must be released under a Creative Commons Attribution-Sharealike or Creative Commons Attribute license. (The Creative Commons Zero waiver is okay, too!). Each entrant must only submit content they have created themselves, and all submissions must adhere to the Ubuntu Code of Conduct.

The winners will be featured in the Ubuntu 15.10 release in October!

We’re looking for work in one of three categories:

  • Wallpapers – we’ll choose 11 stunning photographs, but also one illustrative wallpaper that focuses on Ubuntu 15.10’s codename: wily werewolf.
  • Videos – we need to keep it small, so we’re looking for a 1280x720 video of about 30 seconds.
  • Audio – everybody loves music, so a song between 2-5 minutes will rock speakers around the world!

You’re limited to a total of two submissions in any single category.

We’re still cleaning out the cobwebs, and selecting a panel of judges from the Canonical and Ubuntu design teams and the Ubuntu community. So in the next week or two we’ll announce Flickr, Vimeo, and SoundCloud groups where you can submit your entries. But you’ll have plenty of time to prepare before the final submission deadline: September 15th, 2015 at 23:59 UTC.

There are more technical details for each category, so please see the Ubuntu Free Culture Showcase wiki page for the latest information and updates!

Stuart Langridge: Fooling the natives

Planet Ubuntu - Fri, 08/14/2015 - 15:33
“It’s not death if you refuse it.” — The Crow

Over the last two weeks I have, inadvertently, conducted a very enlightening experiment on myself. I don’t think it’s repeatable or replicable — like gods interfering with free will or The Truman Show, the experiment doesn’t work if you know you’re inside it — but it’s rather illustrative.

I’m a Google+ user1. And the Android G+ app has been doing this annoying thing recently2 where if I pull down to get new posts, sometimes it doesn’t work. Opening the side menu and closing it again always fixed it, so I put it down to some sort of bug in the G+ app and waited for it to be fixed. Not very patiently.

Other people reportedly didn’t have the problem, so I just put it down to weirdness and carried on. Well. I’ve just, this moment, realised why. When I first factory reset this phone and then started it up for the first time, I didn’t use the G+ app. I used the G+ website and bookmarked it to my home screen. And then, it seems, I forgot that I’d done it.

The actual app doesn’t have the problem.

This is interesting in two ways.

The first and most obvious conclusion is that the web app isn’t quite as good as the native app. Obvious conclusion to draw, and also obviously correct; it had a bug that the native app didn’t, and one could reasonably make the case that it had that bug because it’s a web app and not native.

But. I used the thing constantly for a couple of weeks without realising it was a web app. At no point other than this bug did it betray that it was anything other than the G+ app. Now I compare the two side by side, there are differences, but I didn’t notice them as being worse, or even different; I just used the app and it never occurred to me to question it. More importantly, at no point did the bug make me think that it was because I was using a web app, or question the nature of the app. Plenty of people will say that they’ve used web apps and found bugs and therefore web apps are bad. But I wonder how much that would get said if those people hadn’t known they were using a web app? I just put it down to an app bug; it never occurred to me that it might be because it’s in JavaScript rather than Java, and at no other point and in no other way did the G+ web app give me cause to suspect or even question that it was anything other than the native app.

Now, this is not always the case, by any stretch of the imagination. The Google team put a lot of effort into their web stuff. Most “web apps” betray their true nature constantly. But this shows it’s basically possible. If you don’t have the easy, lazy fallback of blaming any issues on a thing being of the web, then you don’t necessarily think of that as being a problem itself.

This is encouraging.

  1. until they shut it down, which I hope they don’t do
  2. since I switched phones from my old Nexus 4 with dodgy wifi to the newer Nexus 4 which used to be my daughter’s

Charles Profitt: Logitech T630 a Micro Mouse

Planet Ubuntu - Thu, 08/13/2015 - 19:05

I have been looking for a smaller more portable mouse to go along with my new Dell XPS 13 (9343) and today I bought the Logitech T630. Usually I try mice before buying them, but no local retailer had one on display. I normally use buttons to navigate back, forward and expo in Ubuntu Unity and I always struggle to find a travel mouse that has those extra features.

The Good:

  • Fantastic fit and finish for a mouse
  • Solid and sturdy
  • Excellent touch gestures that work in Ubuntu Unity out of the box
    • forward (two finger swipe left to right)
    • back (two finger swipe right to left)
    • scroll up / down (swipe up and down on the mouse)
    • right and left click
    • Tabbing through Firefox tabs (left edge swipe)
    • Open the Dash (‘Widows Key”) (two finger double tap)
  • Bluetooth connection was consistent and responsive

The Bad:

  • It is small
  • The clicks are loud

I was very impressed with the touch features of the mouse. I was able to use them in Unity to navigate back, forward and navigate through Firefox tabs. I was also able to open the dash using a two finger double tap and mapped ‘button 2″ to expo. These functions made an excellent replacement for the physical buttons on traditional mouse.

In the end the excellent traits of the mouse were not enough to outweigh the fact that it was not comfortable for me to use. Due to the excessive small size I returned the mouse, but for people who can tolerate a smaller mouse this would be an excellent mobile mouse for Ubuntu.


Mattia Migliorini: Persistent XSS Vulnerability in WordPress 4.2.2

Planet Ubuntu - Thu, 08/13/2015 - 06:29

A few months back Sucuri reported a dangerous security vulnerability found in WordPress 4.2.2. The patch has been released with WordPress 4.2.4, which fixed a total of 6 security vulnerabilities.

Details of the vulnerability

The vulnerability has been initially reported by Sucuri to the WordPress security team on May 6th, 2015, and was fixed with the release of WordPress 4.2.4 on August 4th, 2015.

Vulnerability Details Security Risk: Dangerous
Exploitation level: Easy
DREAD Score: 6/10
Vulnerability: Persistent XSS
Patched Version: 4.2.4 Technical Details

The bug was in the preview_theme() function, which is used by WordPress to changes to theme configurations before they go live. This function is hooked to the setup_theme action hook, which is executed every time a theme is to be displayed.

The theme preview is accessible only to logged-in administrators, specifically users with the switch_themes capability. See the screenshot below:

The preview_theme() function.
Credits: Sucuri

When an administrator visits one of the site’s pages with specific $_GET parameters added, this function will not return early, and set the preview_theme_ob_filter() function as ob_start‘s callback function. This function filters all the HTML links using the preview_theme_ob_filter_callback().

The preview_theme_ob_filter() function.
Credits: Sucuri

So to be clear: first ob_start() gathers all of the page’s content, then calls the preview_theme_ob_filter() callback function to grab all HTML link, which will then be passed to the preview_theme_ob_filter_callback() function.

The preview_theme_ob_filter_callback() function.
Credits: Sucuri

This function does a number of checks, but most importantly removes the onclick='' event handlers from link tags.

The issue

The problem is exactly in the preview_theme_ob_filter_callback() function, because the piece of code responsible for removing the onclick='' handlers can be used to actually insert new HTML tag attributes to the HTML link by sending a tag similar to the following in a post comment:

<a href='/wp-admin' title="onclick='" Title='" style="position: absolute;top:0;left:0;with:100%;height:100%;display:block;" onmouseover=alert(1)//'>Test</a>

Where the part in bold is the piece of code that will get removed, accidentally inserting the style and onmouseover attributes. The resulting tag (visible only to logged-in administrators) will be:

<a href='/wp-admin/' title="" style="position:absolute;top:0;left:0;width:100%;height:100%;display:block;" onmouseover=alert(1)//'>Test</a>

This obviously bypasses WordPress’ filters that were in place to disallow javascript execution.

Update, update, update!

As said before, this vulnerability has been patched with WordPress 4.2.4, which is out since August 4th, 2015. All you have to do, if automatic updates don’t work for you, is update your WordPress website to the most recent version.

The post Persistent XSS Vulnerability in WordPress 4.2.2 appeared first on deshack.

Dustin Kirkland: Ubuntu and LXD at ContainerCon 2015

Planet Ubuntu - Thu, 08/13/2015 - 06:28

Canonical is delighted to sponsor ContainerCon 2015, a Linux Foundation event in Seattle next week, August 17-19, 2015. It's quite exciting to see the A-list of sponsors, many of them newcomers to this particular technology, teaming with energy around containers. 
From chroots to BSD Jails and Solaris Zones, the concepts behind containers were established decades ago, and in fact traverse the spectrum of server operating systems. At Canonical, we've been working on containers in Ubuntu for more than half a decade, providing a home and resources for stewardship and maintenance of the upstream Linux Containers (LXC) project since 2010.

Last year, we publicly shared our designs for LXD -- a new stratum on top of LXC that endows the advantages of a traditional hypervisor into the faster, more efficient world of containers.

Those designs are now reality, with the open source Golang code readily available on Github, and Ubuntu packages available in a PPA for all supported releases of Ubuntu, and already in the Ubuntu 15.10 beta development tree. With ease, you can launch your first LXD containers in seconds, following this simple guide.

LXD is a persistent daemon that provides a clean RESTful interface to manage (start, stop, clone, migrate, etc.) any of the containers on a given host.

Hosts running LXD are handily federated into clusters of container hypervisors, and can work as Nova Compute nodes in OpenStack, for example, delivering Infrastructure-as-a-Service cloud technology at lower costs and greater speeds.

Here, LXD and Docker are quite complementary technologies. LXD furnishes a dynamic platform for "system containers" -- containers that behave like physical or virtual machines, supplying all of the functionality of a full operating system (minus the kernel, which is shared with the host). Such "machine containers" are the core of IaaS clouds, where users focus on instances with compute, storage, and networking that behave like traditional datacenter hardware.

LXD runs perfectly well along with Docker, which supplies a framework for "application containers" -- containers that enclose individual processes that often relate to one another as pools of micro services and deliver complex web applications.

Moreover, the Zen of LXD is the fact that the underlying container implementation is actually decoupled from the RESTful API that drives LXD functionality. We are most excited to discuss next week at ContainerCon our work with Microsoft around the LXD RESTful API, as a cross-platform container management layer.

Ben Armstrong, a Principal Program Manager Lead at Microsoft on the core virtualization and container technologies, has this to say:
“As Microsoft is working to bring Windows Server Containers to the world – we are excited to see all the innovation happening across the industry, and have been collaborating with many projects to encourage and foster this environment. Canonical’s LXD project is providing a new way for people to look at and interact with container technologies. Utilizing ‘system containers’ to bring the advantages of container technology to the core of your cloud infrastructure is a great concept. We are looking forward to seeing the results of our engagement with Canonical in this space.” Finally, if you're in Seattle next week, we hope you'll join us for the technical sessions we're leading at ContainerCon 2015, including: "Putting the D in LXD: Migration of Linux Containers", "Container Security - Past, Present, and Future", and "Large Scale Container Management with LXD and OpenStack". Details are below.
Date: Monday, August 17 • 2:20pm - 3:10pm
Title: Large Scale Container Management with LXD and OpenStack
Speaker: Stéphane Graber
Abstracthttp://sched.co/3YK6
Location: Grand Ballroom B
Schedulehttp://sched.co/3YK6 Date: Wednesday, August 19 10:25am-11:15am
Title: Putting the D in LXD: Migration of Linux Containers
Speaker: Tycho Andersen
Abstract: http://sched.co/3YTz
Location: Willow A
Schedule: http://sched.co/3YTzDate: Wednesday, August 19 • 3:00pm - 3:50pm
Title: Container Security - Past, Present and Future
Speaker: Serge Hallyn
Abstract: http://sched.co/3YTl
Location: Ravenna
Schedule: http://sched.co/3YTlCheers,
Dustin

Didier Roche: Ubuntu Make 15.08 with Scala support and a visual studio code hot fix community story

Planet Ubuntu - Wed, 08/12/2015 - 23:25

Here is a little bit of the start of my day:

As usual, I open the Ubuntu Make large test suite running continuously against trunk and latest release package. I saw that since yesterday 7PM CEST Visual Studio Code page changed its syntax and is not downloadable anymore by Ubuntu Make.

Jumping on the github's project page, I saw a couple of bugs opened about it, and as well a pull request to fix this from a new contributor, Vartan Simonian! All this in less than 12 hours of this breakage. I just had to merge it, changing the medium tests and cut a release. Hey community work!

That made my day, it was thus high time to release this new Ubuntu Make 15.08. Notice that we are starting to follow the scheme "YY.MM" which is quite handy for versioning this kind of continously evolving projects.

In addition to this fix, you will notice that Igor Vuk added scala support. Your always fresh-willingness of scala will now be satisfied through Ubuntu Make!

Some other fixes (progress bar out of range by Anton Antonov, new pep8 release issues found) are also part to make this a great release… And we have even more in the pipe! Thanks again to all our Ubuntu Make contributors, this makes working on this project an awesome journey!

As usual, you can get this latest version direcly in Ubuntu Wily, and through its ppa for the 14.04 LTS, 15.05 ubuntu releases.

Our issue tracker is full of ideas and opportunities, and pull requests remain opened for any issues or suggestions! If you want to be the next featured contributor and want to give an hand, you can refer to this post with useful links!

Ubuntu Kernel Team: Kernel Team Meeting Minutes – August 11, 2015

Planet Ubuntu - Wed, 08/12/2015 - 09:38
Meeting Minutes

IRC Log of the meeting.

Meeting minutes.

Agenda

20150811 Meeting Agenda


Release Metrics and Incoming Bugs

Release metrics and incoming bug data can be reviewed at the following link:

  • http://kernel.ubuntu.com/reports/kt-meeting.txt


Status: Wily Development Kernel

We have rebased our Wily master-next branch to the latest upstream
v4.2-rc6 and uploaded to our ~canonical-kernel-team PPA. We are
still fixing up DKMS packages before we proceed uploading to the
archive.
—–
Important upcoming dates:

  • https://wiki.ubuntu.com/WilyWerewolf/ReleaseSchedule
    Thurs Aug 20 – Feature Freeze (~1 week away)
    Thurs Aug 27 – Beta 1 (~2 weeks away)
    Thurs Sep 24 – Fina Beta (~6 weeks away)


Status: CVE’s

The current CVE status can be reviewed at the following link:

  • http://kernel.ubuntu.com/reports/kernel-cves.html


Status: Stable, Security, and Bugfix Kernel Updates – Precise/Trusty/Utopic/Vivid

Status for the main kernels, until today:

  • Precise – Verification & Testing
  • Trusty – Verification & Testing
  • lts-Utopic – Verification & Testing
  • Vivid – Verification & Testing

    Current opened tracking bugs details:

  • http://kernel.ubuntu.com/sru/kernel-sru-workflow.html
    For SRUs, SRU report is a good source of information:
  • http://kernel.ubuntu.com/sru/sru-report.html

    Schedule:

    cycle: 26-Jul through 15-Aug
    ====================================================================
    24-Jul Last day for kernel commits for this cycle
    26-Jul – 01-Aug Kernel prep week.
    02-Aug – 08-Aug Bug verification & Regression testing.
    09-Aug – 15-Aug Regression testing & Release to -updates.


Open Discussion or Questions? Raise your hand to be recognized

No open discussion.

Daniel Pocock: Recording live events like a pro (part 2: video)

Planet Ubuntu - Wed, 08/12/2015 - 07:55

In the first blog in this series, we looked at how to capture audio effectively for a range of different events. While recording audio appears less complicated than video, it remains fundamental to a good recording. For some types of event, like a speech or a debate, you can have quite a bad video with poor lighting and other defects but people will still be able to watch it if the audio is good. Therefore, if you haven't already looked at the previous blog, please do so now.

As mentioned in the earlier blog, many people now have high quality equipment for recording both audio and video and a wide range of opportunities to use it, whether it is a talk at a conference, a wedding or making a Kickstarter video.

The right camera for video recording

The fundamental piece of equipment is the camera itself. You may have a DSLR camera that can record video or maybe you have a proper handheld video camera. The leading DSLR cameras, combined with a good lens, make higher quality recordings than many handheld video cameras.

Unfortunately, although you pay good money to buy a very well engineered DSLR that could easily record many hours of video, most DSLRs are crippled to record a maximum of 30 minutes in one recording. This issue and some workarounds are discussed later in this blog.

If you don't have any camera at all you need to think carefully about which type to buy. If you are only going to use it once you may want to consider renting or borrowing or asking for other people attending the event to bring whatever cameras they have to help make multiple recordings (the crowdsourcing solution). If you are a very keen photographer then you will probably have a preference for a DSLR.

Accessories

Don't just look at the cost of your camera and conclude that is all the budget you need. For professional quality video recording, you will almost certainly need some accessories. You may find they are overpriced at the retail store where you bought your camera, but you still need some of them, so have a look online.


Recording a talk at a free software event with a Nikon D800 on a very basic tripod with Rode VideoMic Pro, headphones (white cable) and external power (black cable)

If you want to capture audio with the camera and record it in the video file (discussed in more detail below), you will need to purchase a microphone that mounts on the camera. The built-in microphones on cameras are often quite bad, even on the most expensive cameras. If you are just using the built-in microphone for reference audio (to help with time synchronization when you combine different audio files with the video later) then the built-in microphone may be acceptable. Camera audio is discussed in more detail below.

If your camera has a headphone socket, get some headphones for listening to the audio.

Make sure you have multiple memory cards. Look carefully at the speed of the memory cards, slow ones are cheaper but they can't keep up with the speed of writing 1080p video. At a minimum, you should aim to buy memory cards that can handle one or two days worth of data for whatever it is you do.

A tripod is essential for most types of video. If you use a particularly heavy camera or lens or if you are outdoors and it may be windy you will need a heavier tripod for stability. For video, it is very useful to have a tripod with a handle for panning left and right but if the camera will be stationary for the whole recording then the handle is not essential.

Carrying at least one spare battery is another smart move. On one visit to the Inca Trail in Peru, we observed another member of our group hiking up and down the Andes with a brand new DSLR that they couldn't use because the battery was flat.

For extended periods of recording, batteries will not be sufficient and you will need to purchase a mains power supply (PSU). These are available for most types of DSLR and video camera. The camera vendors typically design cameras with unusual power sockets so that you can only use a very specific and heavily overpriced PSU from the same company. Don't forget a surge protector too.

There are various smartphone apps that allow you to remotely control the camera from the screen of your phone, such as the qDslrDashboard app. These often give a better preview than the screen built-in to the camera and may even allow you to use the touch screen to focus more quickly on a specific part of the picture. A regular USB cable is not suitable for this type of app, you need to buy a USB On-The-Go (OTG) cable.


Screenshot of qDslrDashboard app on a smartphone, controlling a DSLR camera

If you plan to copy the video from the camera to a computer at the event, you will need to make sure you have a fast memory card reader. The memory card readers in some laptops are quite slow and others can be quite fast so you may not need to buy an extra card reader.

Camera audio

Most cameras, including DSLRs, have a built-in microphone and a socket for connecting an external microphone.

The built-in microphones obtain very poor quality sound. For many events, it is much better to have independent microphones, such as a lapel microphone attached to a smartphone or wireless transmitter. Those solutions are described in part one of this blog series.

Nonetheless, there are still some benefits of capturing audio in the camera. The biggest benefit is the time synchronization: if you have audio recordings in other devices, you will need to align them with the video using post-production software. If the camera recorded an audio stream too, even if the quality is not very good, you can visualize the waveform on screen and use it to align the other audio recordings much more easily and precisely.

If the camera will be very close to the people speaking then it may be acceptable to use a microphone mounted on the camera. This will be convenient for post-production because the audio will be synchronized with the video. It may still not be as good as a lapel microphone though, but the quality of these camera-mounted microphones is still far higher than the built-in microphones. I've been trying the Rode VideoMic Pro, it is definitely better than recording with the built-in microphone on the camera and also better than the built-in microphone on my phone.

One problem that most people encounter is the sound of the lens autofocus mechanism being detected by the microphone. This occurs with both the built-in microphone and any other microphone you mount on the camera. A microphone mounted on top of the camera doesn't detect this noise with the same intensity as the built-in microphone but it is still present in the recordings.

If using a camera-mounted microphone to detect the audio from an audience, you may need to have an omnidirectional microphone. Many camera-mounted microphones are directional and will not detect very much sound from the sides or behind the camera.

When using any type of external microphone with the camera, it is recommend to disable automatic gain control (AGC) in the camera settings and then manually adjust the microphone sensitivity/volume level.

Use headphones

A final word on audio - most good cameras have an audio output socket. Connect headphones and wear them, to make sure you are always capturing audio. Otherwise, if the microphone's battery goes flat or if a wireless microphone goes out of range you may not notice.

Choosing a lens

The more light you get, the better. Bigger and more expensive lenses allow more light into the camera. Many of the normal lenses sold with a DSLR camera are acceptable but if it is a special occasion you may want to rent a more expensive lens for the day.

If you already have a lens, it is a very good idea to test it in conditions similar to those you expect for the event you want to record.

Recording duration limits

Most DSLR cameras with video capability impose a 30 minute maximum recording duration

This is basically the result of a friendly arrangement between movie studios and politicians to charge an extra tax on video recording technology and potentially make the movie studio bosses richer, supposedly justified by the fact that a tiny but exaggerated number of people use their cameras to record movies at the cinema. As a consequence, most DSLR manufacturers limit the duration of video recording so their product won't be subject to the tax, ensuring the retail price is lower and more attractive.

On top of this, many DSLR cameras also have a 4GB file size limit if they use the FAT filesystem. Recording 1080p video at a high frame rate may hit the file size limit in 10 minutes, well before you encounter the 30 minute maximum recording duration.

To deal with the file size issue, you can record at 720p instead of 1080p and use the frame rate 24fps.

For events longer than 30 minutes or where you really want 1080p or a higher frame rate, there are some other options you can consider:

  • Buy or rent a proper video camera instead of using a DSLR camera
  • Using multiple cameras that can stop and be restarted at different times.
  • Manually stopping and restarting the camera if there are breaks in the event where it is safe to do so.
  • Use an app to control the camera and program it to immediately restart the recording each time it stops
  • Extract the raw output from the camera's HDMI socket and record into some other device or computer. There are several purpose-built devices
    that can be used this way with an embedded SSD for storage.
  • There are also some people distributing unofficial/alternative firmware images that remove the artificial 30 minute recording limit.
Camera settings

There are many online tutorials and demonstration videos on YouTube that will help you optimize the camera settings for video.

You may have already made recordings using the automatic mode. Adjusting some or all of the settings manually may help you create a more optimal recording. You will need to spend some time familiarizing yourself with the settings first.

The first thing to check is white balance. This tells the camera the type of lighting in the location. If you set this incorrectly then the colours will be distorted. Many cameras have the ability to automatically set the white balance.

For video, you may be able to change one or more settings that control the recording quality. These settings control the file compression ratio and image size. Typical image size settings are 720p and 1080p. Compression ratio may be controlled by a high/medium/low quality setting. Choosing the highest quality and biggest picture requires more space on the memory card and also means you reach the 4GB file size limit more quickly. A higher quality setting also implies a faster memory card is required, because the rate of megabytes per second written to the memory card is higher.

Next you need to think about the frame rate. Events that involve fast moving subjects, such as sports, typically benefit from a higher frame rate. For other events it is quite acceptable to use 24 frames per second (fps). Higher frame rates also imply bigger file size and a requirement for a faster memory card.

Once you have decided on the frame rate, the next thing to do is set the shutter speed. Use a shutter speed that is double the frame rate. For example, if using 24fps or 25fps, use a 1/50 shutter speed.

The final two settings you need to adjust are the ISO and aperture. Set these based on the lighting conditions and extent to which the subjects are moving. For example, if the setting is dark or if you are trying to record fast moving subjects like athletes, vehicles or animals, use an ISO value of 800 or higher. Once you have chosen ISO, adjust the aperture to ensure the picture is sufficiently illuminated. Aperture also has a significant impact on the depth of field.

Operating the camera: zoom and focus

Many people use zoom lenses. It is not always necessary to change the zoom while recording a video, you can use software to zoom in and out on individual parts of the picture when editing it in post-production. If you do change the zoom while recording, it may be more difficult to maintain focus.

Almost all lenses support manual focus (turning the focus ring by hand) and many support automatic focus.

When shooting photographs with a DSLR, the mirror is down and the camera can use dedicated sensors for focus and light sensing.

When shooting video, the mirror is up and the camera can not use the same focus sensors that are used in photography. Video recording uses a digital focussing algorithm based on contrast in the picture. If you take a lot of photos you are probably quite accustomed to the fast and precise autofocus for photography and you will notice that keeping a video in focus is more challenging.

As mentioned already, one of the first things you can do to keep focus simple is to avoid zooming while recording. Record in a higher resolution than you require and then zoom with software later. Some people record using 4k resolution even when they only want to produce a 720p video, as they can digitally zoom in to different parts of the 4k recording without losing detail.

If the subject is very stationary (people sitting at a desk for an interview is a typical example), you may be able to set the camera to manual focus and not change it at all while recording.

If you choose to enable autofocus while recording, any built-in camera microphone or microphone mounted on the camera is likely to detect sounds from the motorized focus system.

Ultimately, the autofocus mechanism is not accurate for all subjects and you may be unable to stop them moving around so you will need to change the focus manually while recording. It requires some practice to be able to do this quickly without overshooting the right focus. To make life more tricky, Nikon and Canon focus rings rotate in the opposite direction, so if you are proficient using one brand you may feel awkward if you ever have to use the other. A good way to practice this skill is to practice while in the car or on the train, pointing at different subjects outside the window and trying to stay in focus as you move from one subject to the next.

Make a trial run

Many events, from weddings right up to the Olympic Games opening ceremony, have a trial run the day before. One reason for that is to test the locations and settings of all the recording and broadcasting equipment.

If a trial run isn't possible for your event, you may find some similar event to practice recording and test your equipment. For example, if you are planning to record a wedding, you could try and record a Sunday mass in the same church.

Backup and duplicate the recordings before leaving the event

If you only have one copy of the recordings and the equipment is stolen or damaged you may be very disappointed. Before your event, make a plan to duplicate the raw audio and video recordings so that several people can take copies away with them. Decide in advance who will be responsible for this, ensure there will be several portable hard disks and estimate how much time it will take to prepare the copies and factor this into the schedule.

Conclusion

All the products described can be easily purchased from online retailers. You may not need every accessory that is mentioned as it depends on the type of event you record. The total cost of buying or renting the necessary accessories may be as much as the cost of the camera itself so if you are new to this you may need to think carefully about making a budget with a spreadsheet to do it correctly.

Becoming familiar with the camera controls and practicing the techniques for manual focus and zoom can take weeks or months. If you enjoy photography this can be time well spent but if you don't enjoy it then you may not want to commit the time necessary to make good quality video.

Don't rule out options like renting equipment instead of buying it or crowdsourcing, asking several participants or friends to help make recordings with their own equipment.

For many events, audio is far more indispensable than video and as emphasizing at the beginning of this article, it is recommended that you should be one hundred percent confident in your strategy for recording audio before you start planning to record video.

Daniel Pocock: UK wakes up to airport tax and data grab

Planet Ubuntu - Wed, 08/12/2015 - 07:50

The UK has just woken up to the fact that retailers at airports have been playing the VAT system.

The focus of the media coverage has been on the VAT money, failing to give much emphasis to the fact this is also a sophisticated and coordinated attack on personal privacy.

This situation has been obvious to me for years and it doesn't just occur in the UK. Whenever one of these retailers asks me for a boarding pass, I always refuse. Sometimes they lie to me and tell me that it is mandatory to let them scan my boarding pass: I tell them that a travel companion has the boarding passes and at this point they usually find some way to complete the transaction without further comment.

It is only necessary to pay the correct amount

With the rise of payment cards, people seem to be forgetting that you can pay for things in cash. If I have the right change to pay for something in this scenario, I typically put it on the counter and walk away. Why should I have to waste my time helping a poorly trained cashier understand VAT? I already deal with enough computer problems in the job I'm paid to do, why should I lose my time when on vacation explaining to a cashier that their computer is failing to correctly deduct 20% VAT?

Whenever showing a boarding pass or passport

It is far too common for people to try and scan or copy documents these days. When somebody in an airport shop or hotel asks to see a document, I don't let it out of my hands. It is not hard to cover the barcode with your fingers too. This prevents them making an unauthorized copy or using a barcode scanner to extract personal data. Some of these staff literally try to snatch the documents out of people's hands and drop them onto a scanner next to their till. In some countries hotels are obliged to look at your passport and make a record of the name but they often have no obligation to photocopy or scan it and it is often a huge risk if they do.

If the airports are genuinely concerned about the security of passengers, they would be just as thorough in protecting data as they are in the hunt for terrorists. For example, they could give VAT-free passengers colour-coded boarding passes or some other vouchers without any personal information on them.

VAT diversion funds customer data grab

Shops pocket an extra 20% of the price of a product and they condition everybody, staff and customers, to having all customer data expropriated from the boarding pass at the point of sale, even the customers not eligible for a tax refund are unnecessarily having their data hoovered up in this way. The VAT money diverted away from both the tax man and the customer is rolled back into the system to fund the data grab. UK law promises customers significant protection from unlawful use of personal data, so how can these retailers lie to passengers and tell them that scanning the boarding pass is mandatory? Why not ask the Information Commission's Office to check up on this?

Paying the correct amount and walking away

It is not hard for people to add up the amount of VAT included in a price, deduct it themselves, give the correct and exact amount of cash to the cashier and walk away. Just type the price into your smart phone, divide by 1.2 and you can see the amount to pay. For example, if a product costs £24.95, just type 24.95 ÷ 1.2 into the calculator on your phone and you find that you have to pay £20.79.

It may only be necessary to show a boarding pass or ID for purchasing duty-free alcohol or tobacco.

Not just for airports

Carrying a correct amount of cash doesn't just help navigate the data grab in airports. Consider some of the other scenarios:

  • A long queue at the cashier in a petrol station, typically on a Sunday afternoon. Did you ever notice somebody who just walks past the queue and puts the exact change (or a cheque) on the counter and drives away while everybody else is waiting to pay with some payment card and scan their loyalty card?
  • Restaurant in a tourist spot, you receive a bill and notice they have added VAT and service charges or something you didn't ask for like bread. In European countries and Australia, the price next to each dish on the menu is the only price you have to pay, just like the price on the shelf in a shop or the price on a web page. If you have the right change you can just pay that amount and walk away without a discussion. In the US the taxes are added later and some tourist hot spots in Europe try this with English speaking customers, thinking that if they are American they won't complain.
  • Hotel tries to insist on a currency conversion when charging your credit card. Maybe you've already realized that dynamic currency conversion (DCC) used by retailers often adds at least 3% to a purchase but some hotels try to go way beyond this. The booking confirmation page you printed from their web site gives you a total price in one currency, perhaps USD or EUR and they use some arbitrary exchange rate, with a whopping 30% margin or more, to convert to the local currency and ask you to insert your PIN number or sign a credit card authorization. The best answer to this practice is usually to carry banknotes in the currency used for the booking, paying that exact amount in cash and if they try to argue, just keep counting out the bank notes to prove it matches the confirmation page. Hotels only get away with this trick because few people check the rate, even fewer carry sufficient cash and you may not be able to use local cash machines to get the currency specified in your booking. This is a situation I've encountered in eastern Europe and South America where it is common to quote in EUR and USD even if that isn't the domestic currency.

Rafał Cieślak: Prevent full-screen games from minimizing when switching workspaces

Planet Ubuntu - Tue, 08/11/2015 - 14:36

When I play games on my Ubuntu desktop, I like to switch workspaces a lot. For example, when waiting for respawn I will quickly switch to a second workspace to select a different music track, or to write a quick reply on IM. What I find very inconvenient is that a lot of games, by default, will minimize when I switch workspace. Because of that, it takes me more time to return to game – a workspace switch short-cut, and then alt+tab.

It turns out that this is SDL feature, so all games build with SDL will behave this way. However, there is an easy, little known way to disable it. Simply set the following enviromental variable

export SDL_VIDEO_MINIMIZE_ON_FOCUS_LOSS=0

before starting your game. Or, if you dislike this feature as much as I do, you may want to set that variable in your .profile file, or maybe even /etc/environment.

Enjoy flawless workspace switching when gaming!


Filed under: Ubuntu

Lubuntu Blog: Get Facebook on Pidgin

Planet Ubuntu - Tue, 08/11/2015 - 09:04

Facebook made some changes in his API, so any account trying to access from Pidgin Instant Messenger was unsuccessful, until now, thanks to James Geboski and his new plugin for (L)Ubuntu, Debian and OpenSuse. To install put these five command lines in your terminal:


sudo sh -c "echo 'deb http://download.opensuse.org/repositories/home:/jgeboski/xUbuntu_$(lsb_release -rs)/ /' >> /etc/apt/sources.list.d/jgeboski.list" c

cd /tmp && wget  http://download.opensuse.org/repositories/home:/jgeboski/xUbuntu_$(lsb_release -rs)/Release.key

sudo apt-key add - < Release.key

sudo apt-get update

sudo apt-get install purple-facebook

    Now open Pidgin, create a new Facebook account (important, not a Facebook XMPP one). This plugin will keep up to date, as the developer will upgrade it. Maybe further versions of Lubuntu or Pidgin will ship this by default. Enjoy!

    Daniel Holbach: Ubuntu Community Q&A

    Planet Ubuntu - Tue, 08/11/2015 - 06:49

    If you haven’t heard of it yet, every Tuesday we have the Ubuntu Community Q&A session at 15:00 UTC. It’s always up on http://ubuntuonair.com and you can watch old sessions on the youtube channel. For the casual Ubuntu users it’s a great way to get to know people who are working in the inner circles of Ubuntu and can answer questions, clear up misunderstandings or get specialists on the show.

    Since Jono went to XPRIZE, our team at Canonical has been running them and I really enjoy these sessions. What I liked even more were the sessions where we had guests and got to talk about some more specific topics. In the past few weeks we had Olli Ries on, quite a few UbuCon organisers, some testing/QA heroes and many more.

    If you have anyone you’d like to see interviewed or any specific topics you’d like to see covered, please drop a comment below and we’ll do our best to get them on in the next weeks!

    The Fridge: Ubuntu Weekly Newsletter Issue 429

    Planet Ubuntu - Mon, 08/10/2015 - 22:22

    Pages

    Subscribe to Ubuntu Arizona LoCo Team aggregator