Migrate attachments in background without index (#2208)

- [x] Implement batch migration of attachments without index, i.e. use default
      primary key index on `Message::id`.
- [x] Run attachment in background without index.
- [x] Prepare module for whole database migrations in the future. Once we enable
      that, we have to force (remaining) attachment migration upon startup.
- [x] Run migrations explicitly on startup and remove implicit migrations from
      Backbone models using a placeholder that throws an error.
- [x] `Signal.Debug`: Add support for generating real-world data for
      benchmarking based on contents in `fixtures` folder. Add additional files
      to create a larger variety of test cases, e.g. JPEG, PNG, GIF, MP4, TXT,
      etc. **Test command:**
      ```
      Signal.Debug.createConversation({
        ConversationController,
        WhisperMessage: Whisper.Message,
        numMessages: 100,
      });
      ```
- [x] Minor: Improve error message for `storage.fetch` failures.
- [x] Minor: Use ISO-8601 timestamp for key rotation (helped me debug an issue).
- [x] Update tests to explicitly run migrations.

.eslintignore

@@ -28,4 +28,5 @@ test/views/*.js
 !js/views/message_view.js
 !js/views/settings_view.js
 !main.js
+!preload.js
 !prepare_build.js

app/attachments.js

@@ -24,10 +24,10 @@ exports.ensureDirectory = async (userDataPath) => {
   await fse.ensureDir(exports.getPath(userDataPath));
 };
 
-//      readData :: AttachmentsPath ->
+//      createReader :: AttachmentsPath ->
-//                  RelativePath ->
+//                      RelativePath ->
-//                  IO (Promise ArrayBuffer)
+//                      IO (Promise ArrayBuffer)
-exports.readData = (root) => {
+exports.createReader = (root) => {
   if (!isString(root)) {
     throw new TypeError('`root` must be a path');
   }
@@ -43,10 +43,10 @@ exports.readData = (root) => {
   };
 };
 
-//      writeData :: AttachmentsPath ->
+//      createWriter :: AttachmentsPath ->
-//                   ArrayBuffer ->
+//                      ArrayBuffer ->
-//                   IO (Promise RelativePath)
+//                      IO (Promise RelativePath)
-exports.writeData = (root) => {
+exports.createWriter = (root) => {
   if (!isString(root)) {
     throw new TypeError('`root` must be a path');
   }
@@ -66,8 +66,10 @@ exports.writeData = (root) => {
   };
 };
 
-//      deleteData :: AttachmentsPath -> IO Unit
+//      createDeleter :: AttachmentsPath ->
-exports.deleteData = (root) => {
+//                       RelativePath ->
+//                       IO Unit
+exports.createDeleter = (root) => {
   if (!isString(root)) {
     throw new TypeError('`root` must be a path');
   }

fixtures/README.md

@@ -0,0 +1,4 @@
+A collection of files for generating attachments for load testing. These files
+were made available in the public domain.
+
+Add more files to this directory for `Signal.Debug` to pick up.

fixtures/lorem-ipsum.txt

@@ -0,0 +1,9 @@
+Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas eu nunc commodo, facilisis odio vel, rhoncus justo. Nullam et diam libero. Vivamus sagittis tincidunt enim maximus viverra. Nulla eget imperdiet nulla. Mauris auctor pulvinar eros id eleifend. In lobortis nisi non ex volutpat consequat. Phasellus condimentum ullamcorper pretium. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas.
+
+Mauris at mi leo. Duis accumsan lacus nec diam vulputate, maximus mollis massa vulputate. Suspendisse eget vehicula arcu. Vestibulum ullamcorper dictum odio, consequat faucibus est aliquet quis. Mauris eget est ullamcorper elit auctor faucibus eu pulvinar leo. Cras pulvinar fringilla elementum. Vestibulum varius, magna at gravida porttitor, nunc diam viverra lacus, eu aliquam nunc metus at odio. Aliquam ac sagittis est. Aliquam ac lorem risus. Sed molestie neque et elementum viverra. Morbi laoreet aliquet nisi quis congue. Suspendisse id purus quis sem semper finibus.
+
+Etiam porttitor nisi eu fermentum pellentesque. Phasellus consectetur id nisl eget aliquam. Sed varius at dolor nec feugiat. Curabitur ullamcorper erat eros, vitae fermentum augue hendrerit quis. Pellentesque sed est eleifend, efficitur diam vitae, interdum ligula. Maecenas eleifend ullamcorper ante, eget dictum odio eleifend in. Phasellus non velit non elit fringilla dignissim nec a lectus. Vivamus ornare sagittis risus eget sodales. Vestibulum vestibulum, dolor at viverra hendrerit, dolor ligula feugiat ligula, ut viverra neque ipsum et eros. Suspendisse sed diam sed diam lacinia maximus. Cras consectetur tortor vitae nisi aliquam venenatis. Pellentesque feugiat magna vel pharetra blandit. Nam mollis mattis malesuada.
+
+Morbi dolor dui, efficitur non turpis in, suscipit pulvinar ante. Quisque pretium nisl id blandit ultrices. In molestie libero velit, sit amet pretium ligula auctor eu. Pellentesque ac dui in mi condimentum eleifend et in diam. Sed a elit mattis, pulvinar lacus at, interdum lacus. Proin facilisis nisl magna, nec placerat nulla faucibus at. Aenean aliquet finibus vestibulum. Integer nec massa ligula. Cras finibus vel risus nec ullamcorper. Donec cursus, ante at aliquet venenatis, mi justo rhoncus dui, quis pretium sapien sapien vel nibh. Donec venenatis enim non est efficitur sodales. Duis laoreet pharetra eros at vestibulum. Suspendisse erat sapien, mattis quis risus eget, viverra tempor eros. Nulla gravida, est nec pellentesque porttitor, ipsum mi pellentesque est, sed iaculis tellus velit vitae mi. Curabitur fringilla tortor et erat congue eleifend.
+
+Duis quis vehicula nulla, at consectetur lacus. Praesent non accumsan turpis, vitae pretium eros. Nunc non velit ultrices, dictum massa tempor, faucibus metus. Vestibulum ut eros est. Vestibulum a blandit felis. Nulla iaculis quam sit amet elit gravida dictum. Suspendisse dictum risus a lacus mattis, in hendrerit tortor dictum. Pellentesque pharetra quis ligula a sagittis. Donec quis neque et neque aliquam scelerisque in nec sem. Integer in cursus eros. Quisque pulvinar nunc quis orci consectetur tristique.

js/background.js

@@ -19,7 +19,7 @@
     const { upgradeMessageSchema } = window.Signal.Migrations;
     const {
         Migrations0DatabaseWithAttachmentData,
-        // Migrations1DatabaseWithoutAttachmentData,
+        Migrations1DatabaseWithoutAttachmentData,
     } = window.Signal.Migrations;
     const { Views } = window.Signal;
 
@@ -83,40 +83,25 @@
   const cancelInitializationMessage = Views.Initialization.setMessage();
   console.log('Start IndexedDB migrations');
 
-  console.log('Migrate database with attachments');
+  console.log('Run migrations on database with attachment data');
   await Migrations0DatabaseWithAttachmentData.run({ Backbone });
 
-  // console.log('Migrate attachments to disk');
-  // const database = Migrations0DatabaseWithAttachmentData.getDatabase();
-  // await MessageDataMigrator.processAll({
-  //   Backbone,
-  //   databaseName: database.name,
-  //   minDatabaseVersion: database.version,
-  //   upgradeMessageSchema,
-  // });
-
-  // console.log('Migrate database without attachments');
-  // await Migrations1DatabaseWithoutAttachmentData.run({
-  //   Backbone,
-  //   database: Whisper.Database,
-  // });
-
   console.log('Storage fetch');
   storage.fetch();
 
   const idleDetector = new IdleDetector();
-
-  const NUM_MESSAGE_UPGRADES_PER_IDLE = 2;
   idleDetector.on('idle', async () => {
-    const results = await MessageDataMigrator.processNext({
+    const NUM_MESSAGES_PER_BATCH = 1;
-      BackboneMessage: Whisper.Message,
+    const database = Migrations0DatabaseWithAttachmentData.getDatabase();
-      BackboneMessageCollection: Whisper.MessageCollection,
+    const batch = await MessageDataMigrator.processNextBatchWithoutIndex({
-      count: NUM_MESSAGE_UPGRADES_PER_IDLE,
+      databaseName: database.name,
+      minDatabaseVersion: database.version,
+      numMessagesPerBatch: NUM_MESSAGES_PER_BATCH,
       upgradeMessageSchema,
     });
-    console.log('Upgrade message schema:', results);
+    console.log('Upgrade message schema:', batch);
 
-    if (!results.hasMore) {
+    if (batch.done) {
       idleDetector.stop();
     }
   });

js/database.js

@@ -6,7 +6,7 @@
 (function () {
   'use strict';
 
-  const { Migrations0DatabaseWithAttachmentData } = window.Signal.Migrations;
+  const { getPlaceholderMigrations } = window.Signal.Migrations;
 
   window.Whisper = window.Whisper || {};
   window.Whisper.Database = window.Whisper.Database || {};
@@ -123,5 +123,5 @@
       request.onsuccess = resolve;
     }));
 
-  Whisper.Database.migrations = Migrations0DatabaseWithAttachmentData.migrations;
+  Whisper.Database.migrations = getPlaceholderMigrations();
 }());

js/modules/database.js

@@ -4,18 +4,31 @@
 // and using promises. Revisit use of `idb` dependency as it might cover
 // this functionality.
 
-const { isObject } = require('lodash');
+const { isObject, isNumber } = require('lodash');
 
 
-exports.open = (name, version) => {
+exports.open = (name, version, { onUpgradeNeeded } = {}) => {
   const request = indexedDB.open(name, version);
   return new Promise((resolve, reject) => {
     request.onblocked = () =>
       reject(new Error('Database blocked'));
 
-    request.onupgradeneeded = event =>
+    request.onupgradeneeded = (event) => {
-      reject(new Error('Unexpected database upgrade required:' +
+      const hasRequestedSpecificVersion = isNumber(version);
-        `oldVersion: ${event.oldVersion}, newVersion: ${event.newVersion}`));
+      if (!hasRequestedSpecificVersion) {
+        return;
+      }
+
+      const { newVersion, oldVersion } = event;
+      if (onUpgradeNeeded) {
+        const { transaction } = event.target;
+        onUpgradeNeeded({ oldVersion, transaction });
+        return;
+      }
+
+      reject(new Error('Database upgrade required:' +
+        ` oldVersion: ${oldVersion}, newVersion: ${newVersion}`));
+    };
 
     request.onerror = event =>
       reject(event.target.error);

js/modules/debug.js

@@ -1,3 +1,8 @@
+/* eslint-env node */
+
+const fs = require('fs-extra');
+const path = require('path');
+
 const {
   isFunction,
   isNumber,
@@ -8,6 +13,7 @@ const {
   sample,
 } = require('lodash');
 
+const Attachments = require('../../app/attachments');
 const Message = require('./types/message');
 const { deferredToPromise } = require('./deferred_to_promise');
 const { sleep } = require('./sleep');
@@ -47,7 +53,8 @@ exports.createConversation = async ({
   await Promise.all(range(0, numMessages).map(async (index) => {
     await sleep(index * 100);
     console.log(`Create message ${index + 1}`);
-    const message = new WhisperMessage(createRandomMessage({ conversationId }));
+    const messageAttributes = await createRandomMessage({ conversationId });
+    const message = new WhisperMessage(messageAttributes);
     return deferredToPromise(message.save());
   }));
 };
@@ -71,7 +78,7 @@ const SAMPLE_MESSAGES = [
 ];
 
 const ATTACHMENT_SAMPLE_RATE = 0.33;
-const createRandomMessage = ({ conversationId } = {}) => {
+const createRandomMessage = async ({ conversationId } = {}) => {
   if (!isString(conversationId)) {
     throw new TypeError('"conversationId" must be a string');
   }
@@ -81,7 +88,7 @@ const createRandomMessage = ({ conversationId } = {}) => {
 
   const hasAttachment = Math.random() <= ATTACHMENT_SAMPLE_RATE;
   const attachments = hasAttachment
-    ? [createRandomInMemoryAttachment()] : [];
+    ? [await createRandomInMemoryAttachment()] : [];
   const type = sample(['incoming', 'outgoing']);
   const commonProperties = {
     attachments,
@@ -119,17 +126,40 @@ const _createMessage = ({ commonProperties, conversationId, type } = {}) => {
   }
 };
 
-const MEGA_BYTE = 1e6;
+const FIXTURES_PATH = path.join(__dirname, '..', '..', 'fixtures');
-const createRandomInMemoryAttachment = () => {
+const readData = Attachments.createReader(FIXTURES_PATH);
-  const numBytes = (1 + Math.ceil((Math.random() * 50))) * MEGA_BYTE;
+const createRandomInMemoryAttachment = async () => {
-  const array = new Uint32Array(numBytes).fill(1);
+  const files = (await fs.readdir(FIXTURES_PATH)).map(createFileEntry);
-  const data = array.buffer;
+  const { contentType, fileName } = sample(files);
-  const fileName = Math.random().toString().slice(2);
+  const data = await readData(fileName);
 
   return {
-    contentType: 'application/octet-stream',
+    contentType,
     data,
     fileName,
-    size: numBytes,
+    size: data.byteLength,
   };
 };
+
+const createFileEntry = fileName => ({
+  fileName,
+  contentType: fileNameToContentType(fileName),
+});
+const fileNameToContentType = (fileName) => {
+  const fileExtension = path.extname(fileName).toLowerCase();
+  switch (fileExtension) {
+    case '.gif':
+      return 'image/gif';
+    case '.png':
+      return 'image/png';
+    case '.jpg':
+    case '.jpeg':
+      return 'image/jpeg';
+    case '.mp4':
+      return 'video/mp4';
+    case '.txt':
+      return 'text/plain';
+    default:
+      return 'application/octet-stream';
+  }
+};

js/modules/idle_detector.js

@@ -3,8 +3,8 @@
 const EventEmitter = require('events');
 
 
-const POLL_INTERVAL_MS = 30 * 1000;
+const POLL_INTERVAL_MS = 15 * 1000;
-const IDLE_THRESHOLD_MS = 25;
+const IDLE_THRESHOLD_MS = 20;
 
 class IdleDetector extends EventEmitter {
   constructor() {
@@ -14,10 +14,16 @@ class IdleDetector extends EventEmitter {
   }
 
   start() {
+    console.log('Start idle detector');
     this._scheduleNextCallback();
   }
 
   stop() {
+    console.log('Stop idle detector');
+    this._clearScheduledCallbacks();
+  }
+
+  _clearScheduledCallbacks() {
     if (this.handle) {
       cancelIdleCallback(this.handle);
     }
@@ -28,7 +34,7 @@ class IdleDetector extends EventEmitter {
   }
 
   _scheduleNextCallback() {
-    this.stop();
+    this._clearScheduledCallbacks();
     this.handle = window.requestIdleCallback((deadline) => {
       const { didTimeout } = deadline;
       const timeRemaining = deadline.timeRemaining();

js/modules/messages_data_migrator.js

@@ -1,8 +1,8 @@
 // Module to upgrade the schema of messages, e.g. migrate attachments to disk.
-// `processAll` purposely doesn’t rely on our Backbone IndexedDB adapter to
+// `dangerouslyProcessAllWithoutIndex` purposely doesn’t rely on our Backbone
-// prevent automatic migrations. Rather, it uses direct IndexedDB access.
+// IndexedDB adapter to prevent automatic migrations. Rather, it uses direct
-// This includes avoiding usage of `storage` module which uses Backbone under
+// IndexedDB access. This includes avoiding usage of `storage` module which uses
-// the hood.
+// Backbone under the hood.
 
 /* global IDBKeyRange */
 
@@ -21,12 +21,11 @@ const { deferredToPromise } = require('./deferred_to_promise');
 
 
 const MESSAGES_STORE_NAME = 'messages';
-const NUM_MESSAGES_PER_BATCH = 1;
 
 exports.processNext = async ({
   BackboneMessage,
   BackboneMessageCollection,
-  count,
+  numMessagesPerBatch,
   upgradeMessageSchema,
 } = {}) => {
   if (!isFunction(BackboneMessage)) {
@@ -38,8 +37,8 @@ exports.processNext = async ({
       ' constructor is required');
   }
 
-  if (!isNumber(count)) {
+  if (!isNumber(numMessagesPerBatch)) {
-    throw new TypeError('"count" is required');
+    throw new TypeError('"numMessagesPerBatch" is required');
   }
 
   if (!isFunction(upgradeMessageSchema)) {
@@ -50,7 +49,10 @@ exports.processNext = async ({
 
   const fetchStartTime = Date.now();
   const messagesRequiringSchemaUpgrade =
-    await _fetchMessagesRequiringSchemaUpgrade({ BackboneMessageCollection, count });
+    await _fetchMessagesRequiringSchemaUpgrade({
+      BackboneMessageCollection,
+      count: numMessagesPerBatch,
+    });
   const fetchDuration = Date.now() - fetchStartTime;
 
   const upgradeStartTime = Date.now();
@@ -65,9 +67,9 @@ exports.processNext = async ({
 
   const totalDuration = Date.now() - startTime;
   const numProcessed = messagesRequiringSchemaUpgrade.length;
-  const hasMore = numProcessed > 0;
+  const done = numProcessed < numMessagesPerBatch;
   return {
-    hasMore,
+    done,
     numProcessed,
     fetchDuration,
     upgradeDuration,
@@ -76,9 +78,10 @@ exports.processNext = async ({
   };
 };
 
-exports.processAll = async ({
+exports.dangerouslyProcessAllWithoutIndex = async ({
   databaseName,
   minDatabaseVersion,
+  numMessagesPerBatch,
   upgradeMessageSchema,
 } = {}) => {
   if (!isString(databaseName)) {
@@ -89,6 +92,10 @@ exports.processAll = async ({
     throw new TypeError('"minDatabaseVersion" must be a number');
   }
 
+  if (!isNumber(numMessagesPerBatch)) {
+    throw new TypeError('"numMessagesPerBatch" must be a number');
+  }
+
   if (!isFunction(upgradeMessageSchema)) {
     throw new TypeError('"upgradeMessageSchema" is required');
   }
@@ -106,84 +113,30 @@ exports.processAll = async ({
       ` to be at least ${minDatabaseVersion}`);
   }
 
-  const isComplete = await settings.isAttachmentMigrationComplete(connection);
+  // NOTE: Even if we make this async using `then`, requesting `count` on an
-  console.log('Attachment migration status:', isComplete ? 'complete' : 'incomplete');
+  // IndexedDB store blocks all subsequent transactions, so we might as well
-  if (isComplete) {
+  // explicitly wait for it here:
-    return;
+  const numTotalMessages = await _getNumMessages({ connection });
-  }
-
-  let numTotalMessages = null;
-  // eslint-disable-next-line more/no-then
-  getNumMessages({ connection }).then((numMessages) => {
-    numTotalMessages = numMessages;
-  });
 
   const migrationStartTime = Date.now();
-  let unprocessedMessages = [];
+  let numCumulativeMessagesProcessed = 0;
-  let totalMessagesProcessed = 0;
+  // eslint-disable-next-line no-constant-condition
-  do {
+  while (true) {
-    const lastProcessedIndex =
-      // eslint-disable-next-line no-await-in-loop
-      await settings.getAttachmentMigrationLastProcessedIndex(connection);
-
-    const fetchUnprocessedMessagesStartTime = Date.now();
-    unprocessedMessages =
-      // eslint-disable-next-line no-await-in-loop
-      await _dangerouslyFetchMessagesRequiringSchemaUpgradeWithoutIndex({
-        connection,
-        count: NUM_MESSAGES_PER_BATCH,
-        lastIndex: lastProcessedIndex,
-      });
-    const fetchDuration = Date.now() - fetchUnprocessedMessagesStartTime;
-    const numUnprocessedMessages = unprocessedMessages.length;
-
-    if (numUnprocessedMessages === 0) {
-      break;
-    }
-
-    const upgradeStartTime = Date.now();
-    const upgradedMessages =
-      // eslint-disable-next-line no-await-in-loop
-      await Promise.all(unprocessedMessages.map(upgradeMessageSchema));
-    const upgradeDuration = Date.now() - upgradeStartTime;
-
-    const saveMessagesStartTime = Date.now();
-    const transaction = connection.transaction(MESSAGES_STORE_NAME, 'readwrite');
-    const transactionCompletion = database.completeTransaction(transaction);
     // eslint-disable-next-line no-await-in-loop
-    await Promise.all(upgradedMessages.map(_saveMessage({ transaction })));
+    const status = await _processBatch({
-    // eslint-disable-next-line no-await-in-loop
+      connection,
-    await transactionCompletion;
+      numMessagesPerBatch,
-    const saveDuration = Date.now() - saveMessagesStartTime;
+      upgradeMessageSchema,
-
+    });
-    // TODO: Confirm transaction is complete
+    if (status.done) {
-
+      break;
-    const lastMessage = last(upgradedMessages);
-    const newLastProcessedIndex = lastMessage ? lastMessage.id : null;
-    if (newLastProcessedIndex) {
-      // eslint-disable-next-line no-await-in-loop
-      await settings.setAttachmentMigrationLastProcessedIndex(
-        connection,
-        newLastProcessedIndex
-      );
     }
-
+    numCumulativeMessagesProcessed += status.numMessagesProcessed;
-    totalMessagesProcessed += numUnprocessedMessages;
+    console.log('Upgrade message schema:', Object.assign({}, status, {
-    console.log('Upgrade message schema:', {
-      lastProcessedIndex,
-      numUnprocessedMessages,
-      numCumulativeMessagesProcessed: totalMessagesProcessed,
       numTotalMessages,
-      fetchDuration,
+      numCumulativeMessagesProcessed,
-      saveDuration,
+    }));
-      upgradeDuration,
+  }
-      newLastProcessedIndex,
-      targetSchemaVersion: Message.CURRENT_SCHEMA_VERSION,
-    });
-  } while (unprocessedMessages.length > 0);
-
-  await settings.markAttachmentMigrationComplete(connection);
-  await settings.deleteAttachmentMigrationLastProcessedIndex(connection);
 
   console.log('Close database connection');
   connection.close();
@@ -191,10 +144,128 @@ exports.processAll = async ({
   const totalDuration = Date.now() - migrationStartTime;
   console.log('Attachment migration complete:', {
     totalDuration,
-    totalMessagesProcessed,
+    totalMessagesProcessed: numCumulativeMessagesProcessed,
   });
 };
 
+exports.processNextBatchWithoutIndex = async ({
+  databaseName,
+  minDatabaseVersion,
+  numMessagesPerBatch,
+  upgradeMessageSchema,
+} = {}) => {
+  if (!isFunction(upgradeMessageSchema)) {
+    throw new TypeError('"upgradeMessageSchema" is required');
+  }
+
+  const connection = await _getConnection({ databaseName, minDatabaseVersion });
+  const batch = await _processBatch({
+    connection,
+    numMessagesPerBatch,
+    upgradeMessageSchema,
+  });
+  return batch;
+};
+
+// Private API
+const _getConnection = async ({ databaseName, minDatabaseVersion }) => {
+  if (!isString(databaseName)) {
+    throw new TypeError('"databaseName" must be a string');
+  }
+
+  if (!isNumber(minDatabaseVersion)) {
+    throw new TypeError('"minDatabaseVersion" must be a number');
+  }
+
+  const connection = await database.open(databaseName);
+  const databaseVersion = connection.version;
+  const isValidDatabaseVersion = databaseVersion >= minDatabaseVersion;
+  if (!isValidDatabaseVersion) {
+    throw new Error(`Expected database version (${databaseVersion})` +
+      ` to be at least ${minDatabaseVersion}`);
+  }
+
+  return connection;
+};
+
+const _processBatch = async ({
+  connection,
+  numMessagesPerBatch,
+  upgradeMessageSchema,
+} = {}) => {
+  if (!isObject(connection)) {
+    throw new TypeError('"connection" must be a string');
+  }
+
+  if (!isFunction(upgradeMessageSchema)) {
+    throw new TypeError('"upgradeMessageSchema" is required');
+  }
+
+  if (!isNumber(numMessagesPerBatch)) {
+    throw new TypeError('"numMessagesPerBatch" is required');
+  }
+
+  const isAttachmentMigrationComplete =
+    await settings.isAttachmentMigrationComplete(connection);
+  if (isAttachmentMigrationComplete) {
+    return {
+      done: true,
+    };
+  }
+
+  const lastProcessedIndex =
+    await settings.getAttachmentMigrationLastProcessedIndex(connection);
+
+  const fetchUnprocessedMessagesStartTime = Date.now();
+  const unprocessedMessages =
+    await _dangerouslyFetchMessagesRequiringSchemaUpgradeWithoutIndex({
+      connection,
+      count: numMessagesPerBatch,
+      lastIndex: lastProcessedIndex,
+    });
+  const fetchDuration = Date.now() - fetchUnprocessedMessagesStartTime;
+
+  const upgradeStartTime = Date.now();
+  const upgradedMessages =
+    await Promise.all(unprocessedMessages.map(upgradeMessageSchema));
+  const upgradeDuration = Date.now() - upgradeStartTime;
+
+  const saveMessagesStartTime = Date.now();
+  const transaction = connection.transaction(MESSAGES_STORE_NAME, 'readwrite');
+  const transactionCompletion = database.completeTransaction(transaction);
+  await Promise.all(upgradedMessages.map(_saveMessage({ transaction })));
+  await transactionCompletion;
+  const saveDuration = Date.now() - saveMessagesStartTime;
+
+  const numMessagesProcessed = upgradedMessages.length;
+  const done = numMessagesProcessed < numMessagesPerBatch;
+  const lastMessage = last(upgradedMessages);
+  const newLastProcessedIndex = lastMessage ? lastMessage.id : null;
+  if (!done) {
+    await settings.setAttachmentMigrationLastProcessedIndex(
+      connection,
+      newLastProcessedIndex
+    );
+  } else {
+    await settings.markAttachmentMigrationComplete(connection);
+    await settings.deleteAttachmentMigrationLastProcessedIndex(connection);
+  }
+
+  const batchTotalDuration = Date.now() - fetchUnprocessedMessagesStartTime;
+
+  return {
+    batchTotalDuration,
+    done,
+    fetchDuration,
+    lastProcessedIndex,
+    newLastProcessedIndex,
+    numMessagesProcessed,
+    saveDuration,
+    targetSchemaVersion: Message.CURRENT_SCHEMA_VERSION,
+    upgradeDuration,
+  };
+};
+
 const _saveMessageBackbone = ({ BackboneMessage } = {}) => (message) => {
   const backboneMessage = new BackboneMessage(message);
   return deferredToPromise(backboneMessage.save());
@@ -264,19 +335,29 @@ const _dangerouslyFetchMessagesRequiringSchemaUpgradeWithoutIndex =
     const messagesStore = transaction.objectStore(MESSAGES_STORE_NAME);
 
     const excludeLowerBound = true;
-    const query = hasLastIndex
+    const range = hasLastIndex
       ? IDBKeyRange.lowerBound(lastIndex, excludeLowerBound)
       : undefined;
-    const request = messagesStore.getAll(query, count);
     return new Promise((resolve, reject) => {
-      request.onsuccess = event =>
+      const items = [];
-        resolve(event.target.result);
+      const request = messagesStore.openCursor(range);
+      request.onsuccess = (event) => {
+        const cursor = event.target.result;
+        const hasMoreData = Boolean(cursor);
+        if (!hasMoreData || items.length === count) {
+          resolve(items);
+          return;
+        }
+        const item = cursor.value;
+        items.push(item);
+        cursor.continue();
+      };
       request.onerror = event =>
         reject(event.target.error);
     });
   };
 
-const getNumMessages = async ({ connection } = {}) => {
+const _getNumMessages = async ({ connection } = {}) => {
   if (!isObject(connection)) {
     throw new TypeError('"connection" is required');
   }

js/modules/migrations/get_placeholder_migrations.js

@@ -0,0 +1,23 @@
+const Migrations0DatabaseWithAttachmentData =
+  require('./migrations_0_database_with_attachment_data');
+const Migrations1DatabaseWithoutAttachmentData =
+  require('./migrations_1_database_without_attachment_data');
+
+
+exports.getPlaceholderMigrations = () => {
+  const last0MigrationVersion =
+    Migrations0DatabaseWithAttachmentData.getLatestVersion();
+  const last1MigrationVersion =
+    Migrations1DatabaseWithoutAttachmentData.getLatestVersion();
+
+  const lastMigrationVersion = last1MigrationVersion || last0MigrationVersion;
+
+  return [{
+    version: lastMigrationVersion,
+    migrate() {
+      throw new Error('Unexpected invocation of placeholder migration!' +
+        '\n\nMigrations must explicitly be run upon application startup instead' +
+        ' of implicitly via Backbone IndexedDB adapter at any time.');
+    },
+  }];
+};

js/modules/migrations/migrations_0_database_with_attachment_data.js

@@ -1,4 +1,4 @@
-const { last } = require('lodash');
+const { isString, last } = require('lodash');
 
 const { runMigrations } = require('./run_migrations');
 
@@ -8,7 +8,7 @@ const { runMigrations } = require('./run_migrations');
 // any expensive operations, e.g. modifying all messages / attachments, etc., as
 // it may cause out-of-memory errors for users with long histories:
 // https://github.com/signalapp/Signal-Desktop/issues/2163
-exports.migrations = [
+const migrations = [
   {
     version: '12.0',
     migrate(transaction, next) {
@@ -144,13 +144,29 @@ exports.migrations = [
 const database = {
   id: 'signal',
   nolog: true,
-  migrations: exports.migrations,
+  migrations,
 };
 
-exports.run = ({ Backbone } = {}) =>
+exports.run = ({ Backbone, databaseName } = {}) =>
-  runMigrations({ Backbone, database });
+  runMigrations({
+    Backbone,
+    database: Object.assign(
+      {},
+      database,
+      isString(databaseName) ? { id: databaseName } : {}
+    ),
+  });
 
 exports.getDatabase = () => ({
   name: database.id,
-  version: last(exports.migrations).version,
+  version: exports.getLatestVersion(),
 });
+
+exports.getLatestVersion = () => {
+  const lastMigration = last(migrations);
+  if (!lastMigration) {
+    return null;
+  }
+
+  return lastMigration.version;
+};

js/modules/migrations/migrations_1_database_without_attachment_data.js

@@ -1,15 +1,50 @@
+const { last } = require('lodash');
+
+const db = require('../database');
+const settings = require('../settings');
 const { runMigrations } = require('./run_migrations');
 
 
-exports.migrations = [
+// IMPORTANT: Add new migrations that need to traverse entire database, e.g.
+// messages store, below. Whenever we need this, we need to force attachment
+// migration on startup:
+const migrations = [
   // {
-  //   version: 18,
+  //   version: 0,
-  //   async migrate(transaction, next) {
+  //   migrate(transaction, next) {
-  //     console.log('Migration 18');
-  //     console.log('Attachments stored on disk');
   //     next();
   //   },
   // },
 ];
 
-exports.run = runMigrations;
+exports.run = async ({ Backbone, database } = {}) => {
+  const { canRun } = await exports.getStatus({ database });
+  if (!canRun) {
+    throw new Error('Cannot run migrations on database without attachment data');
+  }
+
+  await runMigrations({ Backbone, database });
+};
+
+exports.getStatus = async ({ database } = {}) => {
+  const connection = await db.open(database.id, database.version);
+  const isAttachmentMigrationComplete =
+    await settings.isAttachmentMigrationComplete(connection);
+  const hasMigrations = migrations.length > 0;
+
+  const canRun = isAttachmentMigrationComplete && hasMigrations;
+  return {
+    isAttachmentMigrationComplete,
+    hasMigrations,
+    canRun,
+  };
+};
+
+exports.getLatestVersion = () => {
+  const lastMigration = last(migrations);
+  if (!lastMigration) {
+    return null;
+  }
+
+  return lastMigration.version;
+};

js/rotate_signed_prekey_listener.js

@@ -43,7 +43,10 @@
         var time = storage.get('nextSignedKeyRotationTime', now);
 
         if (scheduledTime !== time || !timeout) {
-            console.log('Next signed key rotation scheduled for', new Date(time));
+            console.log(
+                'Next signed key rotation scheduled for',
+                new Date(time).toISOString()
+            );
         }
 
         scheduledTime = time;

js/storage.js

@@ -62,10 +62,11 @@
         },
 
         fetch: function() {
-            return new Promise(function(resolve) {
+            return new Promise((resolve, reject) => {
-                items.fetch({reset: true}).fail(function() {
+                items.fetch({reset: true})
-                    console.log('Failed to fetch from storage');
+                    .fail(() => reject(new Error('Failed to fetch from storage.' +
-                }).always(resolve);
+                        ' This may be due to an unexpected database version.')))
+                    .always(resolve);
             });
         },
 

package.json

@@ -12,7 +12,7 @@
   "main": "main.js",
   "scripts": {
     "postinstall": "electron-builder install-app-deps && rimraf node_modules/dtrace-provider",
-    "test": "npm run eslint && npm run test-server && grunt test && npm run test-app && npm run test-modules",
+    "test": "yarn eslint && yarn test-server && grunt test && yarn test-app && yarn test-modules",
     "lint": "grunt jshint",
     "start": "electron .",
     "asarl": "asar l release/mac/Signal.app/Contents/Resources/app.asar",

preload.js

@@ -1,156 +1,168 @@
-(function () {
+/* global Whisper: false */
-  'use strict';
+/* global window: false */
-
+
-  console.log('preload');
+console.log('preload');
-  const electron = require('electron');
+
-
+const electron = require('electron');
-  const Attachment = require('./js/modules/types/attachment');
+
-  const Attachments = require('./app/attachments');
+const Attachment = require('./js/modules/types/attachment');
-  const Message = require('./js/modules/types/message');
+const Attachments = require('./app/attachments');
-  const { deferredToPromise } = require('./js/modules/deferred_to_promise');
+const Message = require('./js/modules/types/message');
-
+const { deferredToPromise } = require('./js/modules/deferred_to_promise');
-  const { app } = electron.remote;
+
-
+const { app } = electron.remote;
-
+
-  window.PROTO_ROOT = 'protos';
+
-  window.config = require('url').parse(window.location.toString(), true).query;
+window.PROTO_ROOT = 'protos';
-  window.wrapDeferred = deferredToPromise;
+window.config = require('url').parse(window.location.toString(), true).query;
-
+
-  const ipc = electron.ipcRenderer;
+window.wrapDeferred = deferredToPromise;
-  window.config.localeMessages = ipc.sendSync('locale-data');
+
-
+const ipc = electron.ipcRenderer;
-  window.setBadgeCount = function(count) {
+window.config.localeMessages = ipc.sendSync('locale-data');
-    ipc.send('set-badge-count', count);
+
-  };
+window.setBadgeCount = count =>
-  window.drawAttention = function() {
+  ipc.send('set-badge-count', count);
-    console.log('draw attention');
+
-    ipc.send('draw-attention');
+window.drawAttention = () => {
-  };
+  console.log('draw attention');
-  window.showWindow = function() {
+  ipc.send('draw-attention');
-    console.log('show window');
+};
-    ipc.send('show-window');
+window.showWindow = () => {
-  };
+  console.log('show window');
-  window.setAutoHideMenuBar = function(autoHide) {
+  ipc.send('show-window');
-    ipc.send('set-auto-hide-menu-bar', autoHide);
+};
-  };
+
-  window.setMenuBarVisibility = function(visibility) {
+window.setAutoHideMenuBar = autoHide =>
-    ipc.send('set-menu-bar-visibility', visibility);
+  ipc.send('set-auto-hide-menu-bar', autoHide);
-  };
+
-  window.restart = function() {
+window.setMenuBarVisibility = visibility =>
-    console.log('restart');
+  ipc.send('set-menu-bar-visibility', visibility);
-    ipc.send('restart');
+
-  };
+window.restart = () => {
-  window.closeAbout = function() {
+  console.log('restart');
-    ipc.send('close-about');
+  ipc.send('restart');
-  };
+};
-  window.updateTrayIcon = function(unreadCount) {
+
-    ipc.send('update-tray-icon', unreadCount);
+window.closeAbout = () =>
-  };
+  ipc.send('close-about');
-
+
-  ipc.on('debug-log', function() {
+window.updateTrayIcon = unreadCount =>
-    Whisper.events.trigger('showDebugLog');
+  ipc.send('update-tray-icon', unreadCount);
-  });
+
-
+ipc.on('debug-log', () => {
-  ipc.on('set-up-with-import', function() {
+  Whisper.events.trigger('showDebugLog');
-    Whisper.events.trigger('setupWithImport');
+});
-  });
+
-
+ipc.on('set-up-with-import', () => {
-  ipc.on('set-up-as-new-device', function() {
+  Whisper.events.trigger('setupWithImport');
-    Whisper.events.trigger('setupAsNewDevice');
+});
-  });
+
-
+ipc.on('set-up-as-new-device', () => {
-  ipc.on('set-up-as-standalone', function() {
+  Whisper.events.trigger('setupAsNewDevice');
-    Whisper.events.trigger('setupAsStandalone');
+});
-  });
+
-
+ipc.on('set-up-as-standalone', () => {
-  ipc.on('show-settings', function() {
+  Whisper.events.trigger('setupAsStandalone');
-    Whisper.events.trigger('showSettings');
+});
-  });
+
-
+ipc.on('show-settings', () => {
-  window.addSetupMenuItems = function() {
+  Whisper.events.trigger('showSettings');
-    ipc.send('add-setup-menu-items');
+});
-  }
+
-
+window.addSetupMenuItems = () =>
-  window.removeSetupMenuItems = function() {
+  ipc.send('add-setup-menu-items');
-    ipc.send('remove-setup-menu-items');
+
-  }
+window.removeSetupMenuItems = () =>
-
+  ipc.send('remove-setup-menu-items');
-  // We pull these dependencies in now, from here, because they have Node.js dependencies
+
-
+// We pull these dependencies in now, from here, because they have Node.js dependencies
-  require('./js/logging');
+
-
+require('./js/logging');
-  if (window.config.proxyUrl) {
+
-    console.log('using proxy url', window.config.proxyUrl);
+if (window.config.proxyUrl) {
-  }
+  console.log('using proxy url', window.config.proxyUrl);
-
+}
-  window.nodeSetImmediate = setImmediate;
+
-  window.nodeWebSocket = require("websocket").w3cwebsocket;
+window.nodeSetImmediate = setImmediate;
-
+window.nodeWebSocket = require('websocket').w3cwebsocket;
-  // Linux seems to periodically let the event loop stop, so this is a global workaround
+
-  setInterval(function() {
+// Linux seems to periodically let the event loop stop, so this is a global workaround
-    window.nodeSetImmediate(function() {});
+setInterval(() => {
-  }, 1000);
+  window.nodeSetImmediate(() => {});
-
+}, 1000);
-  window.dataURLToBlobSync = require('blueimp-canvas-to-blob');
+
-  window.loadImage = require('blueimp-load-image');
+const { autoOrientImage } = require('./js/modules/auto_orient_image');
-  window.ProxyAgent = require('proxy-agent');
+
-  window.EmojiConvertor = require('emoji-js');
+window.autoOrientImage = autoOrientImage;
-  window.emojiData = require('emoji-datasource');
+window.dataURLToBlobSync = require('blueimp-canvas-to-blob');
-  window.nodeFetch = require('node-fetch');
+window.EmojiConvertor = require('emoji-js');
-  window.nodeBuffer = Buffer;
+window.emojiData = require('emoji-datasource');
-  window.EmojiPanel = require('emoji-panel');
+window.EmojiPanel = require('emoji-panel');
-  window.libphonenumber = require('google-libphonenumber').PhoneNumberUtil.getInstance();
+window.libphonenumber = require('google-libphonenumber').PhoneNumberUtil.getInstance();
-  window.libphonenumber.PhoneNumberFormat = require('google-libphonenumber').PhoneNumberFormat;
+window.libphonenumber.PhoneNumberFormat =
-  window.nodeNotifier = require('node-notifier');
+  require('google-libphonenumber').PhoneNumberFormat;
-
+window.loadImage = require('blueimp-load-image');
-  const { autoOrientImage } = require('./js/modules/auto_orient_image');
+
-  window.autoOrientImage = autoOrientImage;
+window.nodeBuffer = Buffer;
-
+window.nodeFetch = require('node-fetch');
-  // ES2015+ modules
+window.nodeNotifier = require('node-notifier');
-  const attachmentsPath = Attachments.getPath(app.getPath('userData'));
+window.ProxyAgent = require('proxy-agent');
-  const deleteAttachmentData = Attachments.deleteData(attachmentsPath);
+
-  const readAttachmentData = Attachments.readData(attachmentsPath);
+// ES2015+ modules
-  const writeAttachmentData = Attachments.writeData(attachmentsPath);
+const attachmentsPath = Attachments.getPath(app.getPath('userData'));
-
+const deleteAttachmentData = Attachments.createDeleter(attachmentsPath);
-  // Injected context functions to keep `Message` agnostic from Electron:
+const readAttachmentData = Attachments.createReader(attachmentsPath);
-  const upgradeSchemaContext = {
+const writeAttachmentData = Attachments.createWriter(attachmentsPath);
-    writeAttachmentData,
+
-  };
+// Injected context functions to keep `Message` agnostic from Electron:
-  const upgradeMessageSchema = message =>
+const upgradeSchemaContext = {
-    Message.upgradeSchema(message, upgradeSchemaContext);
+  writeAttachmentData,
-
+};
-  const { IdleDetector} = require('./js/modules/idle_detector');
+const upgradeMessageSchema = message =>
-
+  Message.upgradeSchema(message, upgradeSchemaContext);
-  window.Signal = {};
+
-  window.Signal.Backup = require('./js/modules/backup');
+const { getPlaceholderMigrations } =
-  window.Signal.Crypto = require('./js/modules/crypto');
+  require('./js/modules/migrations/get_placeholder_migrations');
-  window.Signal.Database = require('./js/modules/database');
+const { IdleDetector } = require('./js/modules/idle_detector');
-  window.Signal.Debug = require('./js/modules/debug');
+
-  window.Signal.Logs = require('./js/modules/logs');
+window.Signal = {};
-  window.Signal.Migrations = {};
+window.Signal.Backup = require('./js/modules/backup');
-  window.Signal.Migrations.loadAttachmentData = Attachment.loadData(readAttachmentData);
+window.Signal.Crypto = require('./js/modules/crypto');
-  window.Signal.Migrations.deleteAttachmentData = Attachment.deleteData(deleteAttachmentData);
+window.Signal.Database = require('./js/modules/database');
-  window.Signal.Migrations.upgradeMessageSchema = upgradeMessageSchema;
+window.Signal.Debug = require('./js/modules/debug');
-  window.Signal.Migrations.Migrations0DatabaseWithAttachmentData =
+window.Signal.Logs = require('./js/modules/logs');
-    require('./js/modules/migrations/migrations_0_database_with_attachment_data');
+
-  window.Signal.Migrations.Migrations1DatabaseWithoutAttachmentData =
+window.Signal.Migrations = {};
-    require('./js/modules/migrations/migrations_1_database_without_attachment_data');
+window.Signal.Migrations.deleteAttachmentData =
-  window.Signal.OS = require('./js/modules/os');
+  Attachment.deleteData(deleteAttachmentData);
-  window.Signal.Settings = require('./js/modules/settings');
+window.Signal.Migrations.getPlaceholderMigrations = getPlaceholderMigrations;
-  window.Signal.Types = {};
+window.Signal.Migrations.loadAttachmentData = Attachment.loadData(readAttachmentData);
-  window.Signal.Types.Attachment = Attachment;
+window.Signal.Migrations.Migrations0DatabaseWithAttachmentData =
-  window.Signal.Types.Errors = require('./js/modules/types/errors');
+  require('./js/modules/migrations/migrations_0_database_with_attachment_data');
-  window.Signal.Types.Message = Message;
+window.Signal.Migrations.Migrations1DatabaseWithoutAttachmentData =
-  window.Signal.Types.MIME = require('./js/modules/types/mime');
+  require('./js/modules/migrations/migrations_1_database_without_attachment_data');
-  window.Signal.Types.Settings = require('./js/modules/types/settings');
+
-  window.Signal.Views = {};
+window.Signal.Migrations.upgradeMessageSchema = upgradeMessageSchema;
-  window.Signal.Views.Initialization = require('./js/modules/views/initialization');
+window.Signal.OS = require('./js/modules/os');
-  window.Signal.Workflow = {};
+window.Signal.Settings = require('./js/modules/settings');
-  window.Signal.Workflow.IdleDetector = IdleDetector;
+
-  window.Signal.Workflow.MessageDataMigrator =
+window.Signal.Types = {};
-    require('./js/modules/messages_data_migrator');
+window.Signal.Types.Attachment = Attachment;
-
+window.Signal.Types.Errors = require('./js/modules/types/errors');
-  // We pull this in last, because the native module involved appears to be sensitive to
+
-  //   /tmp mounted as noexec on Linux.
+window.Signal.Types.Message = Message;
-  require('./js/spell_check');
+window.Signal.Types.MIME = require('./js/modules/types/mime');
-})();
+window.Signal.Types.Settings = require('./js/modules/types/settings');
+
+window.Signal.Views = {};
+window.Signal.Views.Initialization = require('./js/modules/views/initialization');
+
+window.Signal.Workflow = {};
+window.Signal.Workflow.IdleDetector = IdleDetector;
+window.Signal.Workflow.MessageDataMigrator =
+  require('./js/modules/messages_data_migrator');
+
+// We pull this in last, because the native module involved appears to be sensitive to
+//   /tmp mounted as noexec on Linux.
+require('./js/spell_check');

test/_test.js

@@ -1,7 +1,4 @@
-/*
+mocha.setup('bdd');
- * vim: ts=4:sw=4:expandtab
- */
-mocha.setup("bdd");
 window.assert = chai.assert;
 window.PROTO_ROOT = '../protos';
 
@@ -69,18 +66,19 @@ before(function(done) {
   idbReq.onsuccess = function() { done(); };
 });
 
-function clearDatabase(done) {
+async function clearDatabase(done) {
-    var convos = new Whisper.ConversationCollection();
+  await Signal.Migrations.Migrations0DatabaseWithAttachmentData.run({
-    return convos.fetch().then(function() {
+    Backbone,
-        convos.destroyAll().then(function() {
+    databaseName: Whisper.Database.id,
-            var messages = new Whisper.MessageCollection();
+  });
-            return messages.fetch().then(function() {
+
-                messages.destroyAll().then(function() {
+  const convos = new Whisper.ConversationCollection();
-                    if (done) {
+  await convos.fetch();
-                      done();
+  await convos.destroyAll();
-                    }
+  const messages = new Whisper.MessageCollection();
-                });
+  await messages.fetch();
-            });
+  await messages.destroyAll();
-        });
+  if (done) {
-    });
+    done();
+  };
 }

test/app/attachments_test.js

@@ -13,7 +13,7 @@ const NAME_LENGTH = 64;
 const PATH_LENGTH = PREFIX_LENGTH + NUM_SEPARATORS + NAME_LENGTH;
 
 describe('Attachments', () => {
-  describe('writeData', () => {
+  describe('createWriter', () => {
     let tempRootDirectory = null;
     before(() => {
       tempRootDirectory = tmp.dirSync().name;
@@ -25,9 +25,9 @@ describe('Attachments', () => {
 
     it('should write file to disk and return path', async () => {
       const input = stringToArrayBuffer('test string');
-      const tempDirectory = path.join(tempRootDirectory, 'Attachments_writeData');
+      const tempDirectory = path.join(tempRootDirectory, 'Attachments_createWriter');
 
-      const outputPath = await Attachments.writeData(tempDirectory)(input);
+      const outputPath = await Attachments.createWriter(tempDirectory)(input);
       const output = await fse.readFile(path.join(tempDirectory, outputPath));
 
       assert.lengthOf(outputPath, PATH_LENGTH);
@@ -37,7 +37,7 @@ describe('Attachments', () => {
     });
   });
 
-  describe('readData', () => {
+  describe('createReader', () => {
     let tempRootDirectory = null;
     before(() => {
       tempRootDirectory = tmp.dirSync().name;
@@ -48,7 +48,7 @@ describe('Attachments', () => {
     });
 
     it('should read file from disk', async () => {
-      const tempDirectory = path.join(tempRootDirectory, 'Attachments_readData');
+      const tempDirectory = path.join(tempRootDirectory, 'Attachments_createReader');
 
       const relativePath = Attachments.getRelativePath(Attachments.createName());
       const fullPath = path.join(tempDirectory, relativePath);
@@ -57,13 +57,13 @@ describe('Attachments', () => {
       const inputBuffer = Buffer.from(input);
       await fse.ensureFile(fullPath);
       await fse.writeFile(fullPath, inputBuffer);
-      const output = await Attachments.readData(tempDirectory)(relativePath);
+      const output = await Attachments.createReader(tempDirectory)(relativePath);
 
       assert.deepEqual(input, output);
     });
   });
 
-  describe('deleteData', () => {
+  describe('createDeleter', () => {
     let tempRootDirectory = null;
     before(() => {
       tempRootDirectory = tmp.dirSync().name;
@@ -74,7 +74,7 @@ describe('Attachments', () => {
     });
 
     it('should delete file from disk', async () => {
-      const tempDirectory = path.join(tempRootDirectory, 'Attachments_deleteData');
+      const tempDirectory = path.join(tempRootDirectory, 'Attachments_createDeleter');
 
       const relativePath = Attachments.getRelativePath(Attachments.createName());
       const fullPath = path.join(tempDirectory, relativePath);
@@ -83,7 +83,7 @@ describe('Attachments', () => {
       const inputBuffer = Buffer.from(input);
       await fse.ensureFile(fullPath);
       await fse.writeFile(fullPath, inputBuffer);
-      await Attachments.deleteData(tempDirectory)(relativePath);
+      await Attachments.createDeleter(tempDirectory)(relativePath);
 
       const existsFile = await fse.exists(fullPath);
       assert.isFalse(existsFile);

test/views/message_view_test.js

@@ -1,17 +1,19 @@
 describe('MessageView', function() {
   var convo, message;
 
-  before(function() {
+  before(async (done) => {
+    await clearDatabase();
     convo = new Whisper.Conversation({id: 'foo'});
     message = convo.messageCollection.add({
       conversationId: convo.id,
       body: 'hello world',
       type: 'outgoing',
       source: '+14158675309',
-      received_at: Date.now()
+      received_at: Date.now(),
     });
 
-    return storage.put('number_id', '+18088888888.1');
+    await storage.put('number_id', '+18088888888.1');
+    done();
   });
 
   it('should display the message text', function() {