Package Details: firefox-syncstorage-git 0.13.0.r10.gab5df9ba-1

Git Clone URL: https://aur.archlinux.org/firefox-syncstorage-git.git (read-only, click to copy)
Package Base: firefox-syncstorage-git
Description: Sync storage with build-in token server for running a self-hosted firefox sync server.
Upstream URL: https://github.com/mozilla-services/syncstorage-rs
Keywords: firefox morzilla sync syncserver
Licenses: MPL2
Conflicts: firefox-syncstorage
Provides: firefox-syncstorage
Submitter: jewelux
Maintainer: jewelux
Last Packager: jewelux
Votes: 10
Popularity: 0.000000
First Submitted: 2021-12-12 18:41 (UTC)
Last Updated: 2023-01-12 18:55 (UTC)

Pinned Comments

jewelux commented on 2021-12-12 19:15 (UTC) (edited on 2023-01-12 18:59 (UTC) by jewelux)

Database

To setup a fresh MySQL DB and user: (mysql -u root):

CREATE USER "sample_user"@"localhost" IDENTIFIED BY "sample_password";
CREATE DATABASE syncstorage_rs;
GRANT ALL PRIVILEGES on syncstorage_rs.* to sample_user@localhost;
Config

Open file /etc/firefox-syncstorage.toml to set your own parameters. Here is an example.

port=5000
syncstorage.database_url = "mysql://sample_user:sample_password@127.0.0.1:3306/syncstorage_rs"
# get secret with: head -c 20 /dev/urandom | sha1sum
master_secret = "my_secret"
human_logs = 1
tokenserver.enabled = true
tokenserver.node_type = "mysql"
tokenserver.database_url = "mysql://sample_user:sample_password@127.0.0.1:3306/syncstorage_rs"
tokenserver.fxa_metrics_hash_secret = "my_secret"
tokenserver.fxa_email_domain = "api.accounts.firefox.com"
tokenserver.fxa_oauth_server_url = "https://oauth.accounts.firefox.com/v1"
cors_allowed_origin = "null"
cors_max_age = 86400
Firefox

In abount:config change tokenserver url to your service url.

identity.sync.tokenserver.uri    https://mydomain.tld/1.0/sync/1.5
Run service

systemctl start firefox-syncstorage.service

Add node record

Currently it is necessary to add your node (incl. service pattern) to tokenserver database.
Your can add the node record by the following sql. Don't forget to replace the example url https://mydomain.tld with your service url. 

INSERT INTO `services` (`id`, `service`, `pattern`) VALUES ('1', 'sync-1.5', '{node}/1.5/{uid}');
INSERT INTO `nodes` (`id`, `service`, `node`, `available`, `current_load`, `capacity`, `downed`, `backoff`) VALUES ('1', '1', 'https://mydomain.tld', '1', '0', '1', '0', '0');
Troubleshooting
Trouble with Apache as reverse proxy

If you use TLS you should modify the request header and set X-Forwarded-Proto to https. Here is an example of a virtual host file:

<VirtualHost *:443>
   ServerName mydomain.tld
   DocumentRoot /srv/http/webservices/
   RequestHeader set X-Forwarded-Proto https
   ProxyPreserveHost On
   ProxyPass / http://127.0.0.1:5000/
   ProxyPassReverse / http://127.0.0.1:5000/
   <Proxy *>
      AllowOverride all
      Require all granted
   </Proxy>
</VirtualHost>
Sync in Firefox Mobile Nightly is broken

Firefox Mobile client expected X-Timestamp in response header.
Here is an example to set X-Timestamp header in apache virtual host file:

Header set X-Timestamp %t
Header edit X-Timestamp t= ""

The issue is fixed and the workaround should not be necessary anymore.

Latest Comments

« First ‹ Previous 1 2 3 4 5 6 Next › Last »

jewelux commented on 2022-03-19 14:40 (UTC) (edited on 2022-03-19 14:44 (UTC) by jewelux)

@lexaiden By now, there are two thinks that are not added by tokenserver automatically.

  1. A service pattern (that is what you did)
  2. Your node with your server-url (Explained in "Troubleshooting"->"Node is not automatically added" in my how-to comment.)

The nodes-service-ID should be the same ID of "sync-1.5" pattern (and not the ID of "sync.1.1" pattern or any other ID that does not exists.)
In both of my sql-examples (Insert-nodes-sql and Insert-services-sql) I used id 2 for "sync-1.5" pattern.
But you should check, if you also used the same id (for example ID 2).

lexaiden commented on 2022-03-19 14:15 (UTC)

@jewelux Thank you so much! The SQL command did the trick, but what did you mean by "node record points to the sync-1.5 pattern"?

jewelux commented on 2022-03-19 12:01 (UTC)

@lexaiden: I think you are missing a service pattern in service table. You can add the patterns with this sql:

INSERT INTO services (id, service, pattern) VALUES
    (1, "sync-1.1", "{node}/1.1/{uid}"),
    (2, "sync-1.5", "{node}/1.5/{uid}");

Take care, that your node record points to the sync-1.5 pattern. Please give me a feedback, when you successfully fixed the issue. I should add it to the how-to.

lexaiden commented on 2022-03-18 21:37 (UTC) (edited on 2022-03-18 21:39 (UTC) by lexaiden)

This does not work for me, the firefox-syncstorgae.service is running, but when a firefox browser tries to sync I get this error:

sudo journalctl -r -u firefox-syncstorage.service

Mär 18 22:20:14 lexserv05 firefox-syncstorage[26635]: , status: 500 }
Mär 18 22:20:14 lexserv05 firefox-syncstorage[26635]:    5: __clone
Mär 18 22:20:14 lexserv05 firefox-syncstorage[26635]:    4: start_thread
Mär 18 22:20:14 lexserv05 firefox-syncstorage[26635]:    3: <unknown>
Mär 18 22:20:14 lexserv05 firefox-syncstorage[26635]:    2: <unknown>
Mär 18 22:20:14 lexserv05 firefox-syncstorage[26635]:    1: <unknown>
Mär 18 22:20:14 lexserv05 firefox-syncstorage[26635]: Mar 18 22:20:14.134 ERRO Internal Server Error: ApiError { kind: Db(DbError { kind: DieselQuery(NotFound), status: 500 }), backtrace:    0: <unknown>
Mär 18 22:20:03 lexserv05 systemd[1]: Started Actix web server running Mozilla Firefox Sync Storage.
Mär 18 22:20:03 lexserv05 systemd[1]: firefox-syncstorage.service: Consumed 1.242s CPU time.
Mär 18 22:20:03 lexserv05 systemd[1]: Stopped Actix web server running Mozilla Firefox Sync Storage.
Mär 18 22:20:03 lexserv05 systemd[1]: firefox-syncstorage.service: Deactivated successfully.
Mär 18 22:20:03 lexserv05 systemd[1]: Stopping Actix web server running Mozilla Firefox Sync Storage...

Does anyone have any idea what is causing the problem? Database export:


-- phpMyAdmin SQL Dump
-- version 5.1.3
-- https://www.phpmyadmin.net/
--
-- Host: localhost
-- Generation Time: Mar 18, 2022 at 10:35 PM
-- Server version: 10.7.3-MariaDB
-- PHP Version: 8.1.3

SET SQL_MODE = "NO_AUTO_VALUE_ON_ZERO";
START TRANSACTION;
SET time_zone = "+00:00";


/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8mb4 */;

--
-- Database: `fxsync_rs`
--

-- --------------------------------------------------------

--
-- Table structure for table `batch_uploads`
--

CREATE TABLE `batch_uploads` (
  `batch` bigint(20) NOT NULL,
  `userid` bigint(20) NOT NULL,
  `collection` int(11) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

-- --------------------------------------------------------

--
-- Table structure for table `batch_upload_items`
--

CREATE TABLE `batch_upload_items` (
  `batch` bigint(20) NOT NULL,
  `userid` bigint(20) NOT NULL,
  `id` varchar(64) NOT NULL,
  `sortindex` int(11) DEFAULT NULL,
  `payload` mediumtext DEFAULT NULL,
  `payload_size` int(11) DEFAULT NULL,
  `ttl_offset` int(11) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

-- --------------------------------------------------------

--
-- Table structure for table `bso`
--

CREATE TABLE `bso` (
  `userid` bigint(20) NOT NULL,
  `collection` int(11) NOT NULL,
  `id` varchar(64) NOT NULL,
  `sortindex` int(11) DEFAULT NULL,
  `payload` mediumtext NOT NULL,
  `modified` bigint(20) NOT NULL,
  `ttl` bigint(20) NOT NULL,
  `payload_size` int(11) DEFAULT 0
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

-- --------------------------------------------------------

--
-- Table structure for table `collections`
--

CREATE TABLE `collections` (
  `id` int(11) NOT NULL,
  `name` varchar(32) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

--
-- Dumping data for table `collections`
--

INSERT INTO `collections` (`id`, `name`) VALUES
(100, ''),
(11, 'addons'),
(12, 'addresses'),
(7, 'bookmarks'),
(1, 'clients'),
(13, 'creditcards'),
(2, 'crypto'),
(3, 'forms'),
(4, 'history'),
(5, 'keys'),
(6, 'meta'),
(10, 'passwords'),
(8, 'prefs'),
(9, 'tabs');

-- --------------------------------------------------------

--
-- Table structure for table `nodes`
--

CREATE TABLE `nodes` (
  `id` bigint(20) NOT NULL,
  `service` int(11) NOT NULL,
  `node` varchar(64) COLLATE utf8mb4_unicode_ci NOT NULL,
  `available` int(11) NOT NULL,
  `current_load` int(11) NOT NULL,
  `capacity` int(11) NOT NULL,
  `downed` int(11) NOT NULL,
  `backoff` int(11) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

-- --------------------------------------------------------

--
-- Table structure for table `services`
--

CREATE TABLE `services` (
  `id` int(11) NOT NULL,
  `service` varchar(30) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `pattern` varchar(128) COLLATE utf8mb4_unicode_ci DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

-- --------------------------------------------------------

--
-- Table structure for table `users`
--

CREATE TABLE `users` (
  `uid` bigint(20) NOT NULL,
  `service` int(11) NOT NULL,
  `email` varchar(255) NOT NULL,
  `generation` bigint(20) NOT NULL,
  `client_state` varchar(32) NOT NULL,
  `created_at` bigint(20) NOT NULL,
  `replaced_at` bigint(20) DEFAULT NULL,
  `nodeid` bigint(20) NOT NULL,
  `keys_changed_at` bigint(20) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3;

-- --------------------------------------------------------

--
-- Table structure for table `user_collections`
--

CREATE TABLE `user_collections` (
  `userid` bigint(20) NOT NULL,
  `collection` int(11) NOT NULL,
  `last_modified` bigint(20) NOT NULL,
  `total_bytes` bigint(20) DEFAULT NULL,
  `count` int(11) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

-- --------------------------------------------------------

--
-- Table structure for table `__diesel_schema_migrations`
--

CREATE TABLE `__diesel_schema_migrations` (
  `version` varchar(50) COLLATE utf8mb4_unicode_ci NOT NULL,
  `run_on` timestamp NOT NULL DEFAULT current_timestamp()
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

--
-- Dumping data for table `__diesel_schema_migrations`
--

INSERT INTO `__diesel_schema_migrations` (`version`, `run_on`) VALUES
('20180828010336', '2022-03-18 21:15:03'),
('20190911164500', '2022-03-18 21:15:03'),
('20190925174347', '2022-03-18 21:15:03'),
('20200403102015', '2022-03-18 21:15:04'),
('20200612231034', '2022-03-18 21:15:04'),
('20200824091401', '2022-03-18 21:15:04'),
('20210716001122', '2022-03-18 21:15:13'),
('20210803234845', '2022-03-18 21:15:13'),
('20210930142643', '2022-03-18 21:15:13'),
('20210930142654', '2022-03-18 21:15:13'),
('20210930142746', '2022-03-18 21:15:13'),
('20210930144043', '2022-03-18 21:15:13'),
('20210930144225', '2022-03-18 21:15:13'),
('20211222160451', '2022-03-18 21:15:13');

--
-- Indexes for dumped tables
--

--
-- Indexes for table `batch_uploads`
--
ALTER TABLE `batch_uploads`
  ADD PRIMARY KEY (`batch`,`userid`);

--
-- Indexes for table `batch_upload_items`
--
ALTER TABLE `batch_upload_items`
  ADD PRIMARY KEY (`batch`,`userid`,`id`);

--
-- Indexes for table `bso`
--
ALTER TABLE `bso`
  ADD PRIMARY KEY (`userid`,`collection`,`id`),
  ADD KEY `bso_expiry_idx` (`ttl`),
  ADD KEY `bso_usr_col_mod_idx` (`userid`,`collection`,`modified`);

--
-- Indexes for table `collections`
--
ALTER TABLE `collections`
  ADD PRIMARY KEY (`id`),
  ADD UNIQUE KEY `name` (`name`);

--
-- Indexes for table `nodes`
--
ALTER TABLE `nodes`
  ADD PRIMARY KEY (`id`),
  ADD UNIQUE KEY `unique_idx` (`service`,`node`);

--
-- Indexes for table `services`
--
ALTER TABLE `services`
  ADD PRIMARY KEY (`id`),
  ADD UNIQUE KEY `service` (`service`);

--
-- Indexes for table `users`
--
ALTER TABLE `users`
  ADD PRIMARY KEY (`uid`),
  ADD KEY `lookup_idx` (`email`,`service`,`created_at`),
  ADD KEY `replaced_at_idx` (`service`,`replaced_at`),
  ADD KEY `node_idx` (`nodeid`);

--
-- Indexes for table `user_collections`
--
ALTER TABLE `user_collections`
  ADD PRIMARY KEY (`userid`,`collection`);

--
-- Indexes for table `__diesel_schema_migrations`
--
ALTER TABLE `__diesel_schema_migrations`
  ADD PRIMARY KEY (`version`);

--
-- AUTO_INCREMENT for dumped tables
--

--
-- AUTO_INCREMENT for table `collections`
--
ALTER TABLE `collections`
  MODIFY `id` int(11) NOT NULL AUTO_INCREMENT, AUTO_INCREMENT=101;

--
-- AUTO_INCREMENT for table `nodes`
--
ALTER TABLE `nodes`
  MODIFY `id` bigint(20) NOT NULL AUTO_INCREMENT;

--
-- AUTO_INCREMENT for table `services`
--
ALTER TABLE `services`
  MODIFY `id` int(11) NOT NULL AUTO_INCREMENT, AUTO_INCREMENT=3;

--
-- AUTO_INCREMENT for table `users`
--
ALTER TABLE `users`
  MODIFY `uid` bigint(20) NOT NULL AUTO_INCREMENT, AUTO_INCREMENT=4;
COMMIT;

/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;

jewelux commented on 2021-12-12 19:15 (UTC) (edited on 2023-01-12 18:59 (UTC) by jewelux)

Database

To setup a fresh MySQL DB and user: (mysql -u root):

CREATE USER "sample_user"@"localhost" IDENTIFIED BY "sample_password";
CREATE DATABASE syncstorage_rs;
GRANT ALL PRIVILEGES on syncstorage_rs.* to sample_user@localhost;
Config

Open file /etc/firefox-syncstorage.toml to set your own parameters. Here is an example.

port=5000
syncstorage.database_url = "mysql://sample_user:sample_password@127.0.0.1:3306/syncstorage_rs"
# get secret with: head -c 20 /dev/urandom | sha1sum
master_secret = "my_secret"
human_logs = 1
tokenserver.enabled = true
tokenserver.node_type = "mysql"
tokenserver.database_url = "mysql://sample_user:sample_password@127.0.0.1:3306/syncstorage_rs"
tokenserver.fxa_metrics_hash_secret = "my_secret"
tokenserver.fxa_email_domain = "api.accounts.firefox.com"
tokenserver.fxa_oauth_server_url = "https://oauth.accounts.firefox.com/v1"
cors_allowed_origin = "null"
cors_max_age = 86400
Firefox

In abount:config change tokenserver url to your service url.

identity.sync.tokenserver.uri    https://mydomain.tld/1.0/sync/1.5
Run service

systemctl start firefox-syncstorage.service

Add node record

Currently it is necessary to add your node (incl. service pattern) to tokenserver database.
Your can add the node record by the following sql. Don't forget to replace the example url https://mydomain.tld with your service url. 

INSERT INTO `services` (`id`, `service`, `pattern`) VALUES ('1', 'sync-1.5', '{node}/1.5/{uid}');
INSERT INTO `nodes` (`id`, `service`, `node`, `available`, `current_load`, `capacity`, `downed`, `backoff`) VALUES ('1', '1', 'https://mydomain.tld', '1', '0', '1', '0', '0');
Troubleshooting
Trouble with Apache as reverse proxy

If you use TLS you should modify the request header and set X-Forwarded-Proto to https. Here is an example of a virtual host file:

<VirtualHost *:443>
   ServerName mydomain.tld
   DocumentRoot /srv/http/webservices/
   RequestHeader set X-Forwarded-Proto https
   ProxyPreserveHost On
   ProxyPass / http://127.0.0.1:5000/
   ProxyPassReverse / http://127.0.0.1:5000/
   <Proxy *>
      AllowOverride all
      Require all granted
   </Proxy>
</VirtualHost>
Sync in Firefox Mobile Nightly is broken

Firefox Mobile client expected X-Timestamp in response header.
Here is an example to set X-Timestamp header in apache virtual host file:

Header set X-Timestamp %t
Header edit X-Timestamp t= ""

The issue is fixed and the workaround should not be necessary anymore.

Siosm commented on 2018-10-23 21:32 (UTC)

Sorry, but I don't use this package anymore. I'm disowning it.

oi_wtf commented on 2018-10-23 10:16 (UTC) (edited on 2018-10-23 10:26 (UTC) by oi_wtf)

This should depend on python2-virtualenv . Otherwise a build in a clean chroot fails with:

==> Starting build()...
`which python2 python | head -n 1` -m virtualenv --python=`which python2 python | head -n 1` --no-site-packages ./local
/usr/bin/python2: No module named virtualenv
make: *** [Makefile:27: local/COMPLETE] Error 1
==> ERROR: A failure occurred in build().

EDIT: Clarification: I'm not sure if maybe both py3 and py2 versions of virtualenv are needed (since building worked a few days ago), I didn't look into the build-process...

mqs commented on 2017-07-27 14:41 (UTC) (edited on 2017-07-27 14:42 (UTC) by mqs)

@Siosm I would also really like it if you could add armv6h and armv7h to the arch array. ArchlinuxARM is not officially supported but so is anything within the AUR. According to https://bbs.archlinux.org/viewtopic.php?id=160737 it's only up to you and I can't see any disadvantages for you, as the package runs on arm exactly the same as on x86_64. Additionally, ArchlinuxARM has its own forum, official repos and wiki, but there is only one AUR so we can't just use another distribution platform... So, please do us the favor and just put those few letters into the PKGBUILD. If you still think you shouldn't do it, then thank you anyway, for making this package available for us, in fact,just adding the architecture to the arch array is not always enough to make it work.

figue commented on 2017-06-30 10:06 (UTC)

Can you add mod_wsgi2 as an optdepend? If you want to run /opt/mozilla-firefox-sync-server/syncserver.wsgi with Apache, you need the package mod_wsgi2 (which is for Python2). Thank you!

Siosm commented on 2017-02-11 12:36 (UTC)

I'm sorry I will not add unofficially supported arch. Please use makepkg -A.

« First ‹ Previous 1 2 3 4 5 6 Next › Last »