Privacy-preserving record linkage in large databases using secure multiparty computation

Cryptographic building blocks

We fix two cryptographic functions: the hash function H:{0,1}^∗→{0,1}^η, and a block cipher \(E:\{0,1\}^{\eta }\times \mathcal {K}\rightarrow \{0,1\}^{\eta }\), where \(\mathcal {K}\) is the set of possible keys for E. The block size of E should be the same as the output length of H, so that we could apply E directly to the output of H. The challenge is that we will apply E to secret-shared values, and hence the block cipher E has to be easily computable under SMC. We have picked AES-128 as E, and the privacy-preserving implementation of AES-128 is already available in Sharemind [19]. There exist newer, possibly more efficient implementations of AES [23], as well as proposals for SMC-optimized block ciphers [24], which have not been implemented on Sharemind yet, and could potentially speed the computation up.

In our solution, we have taken η=128. We let H be the composition of SHA-256 cryptographic hash function and a universal one-way hash function family (UOWHF) [25].

Computation on the client side

The behaviour of a client is described in Algorithm 1. At start-up, each client queries the Sharemind servers for the random parameters of the UOWHF H, which the servers generate in the beginning. Each client takes the input records as soon as they become available, picks the necessary attributes from each record, applies H to them, and uploads the hashes to Sharemind servers in secret-shared manner. The Sharemind API readily supports that. At this point, no server learns the exact values of the hashes, since they are all secret-shared.

In the end, when the servers have finished the computation, the client queries them for its personal result. The servers respond with the shares of the output, which is a vector of booleans of the same length as the number of records from this client, indicating whether a record is a duplicate one. The client reconstructs the result vector. Again, the Sharemind API already has support for that.

Computation on the server side

We describe the work of the servers in phases.

Start-up.

This short phase is given in Algorithm 2. The servers privately generate the following random values:

• parameters of the hash function H;

• a key ⟦K⟧ for the block cipher E.

All these values are generated in such a way that they remain secret-shared among the servers, and no server actually learns them. Sharemind API supports such shared randomness generation, and we denote the corresponding functionality as random.

The servers initialize a public variable cnt ←0. It will be used for indexation of clients.

Upload

During the upload phase, two different activity threads are essentially taking place in parallel.

One thread is the actual acceptance of data from the clients, described in Algorithm 3. The hashes of clients are stored into an private array ⟦v⟧, and the corresponding client identities into a public array s under the same indices. In principle, this algorithm could be invoked several times for the same client, if it intends to split up the upload of its data. Several clients may want to connect to the servers at the same time, so we need to make use of Sharemind’s database support to avoid race condition on cnt.

The other thread of activity is encrypting the elements ⟦v_i⟧. For each i, the servers evaluate E on ⟦v_i⟧ in a privacy-preserving way, using the same key ⟦K⟧ for each encryption, and take the first 64 bits of the result, as described in Algorithm 4. The only reason why we take only 64 of 128 bits is that the largest ring that Sharemind currently supports is \(\mathbb {Z}_{2^{64}}\). There are no problems with taking only 64 bits, as long as there are no collisions. With the envisioned amounts of data (around 10⁷=2^23.5 records in total), collisions are unlikely; the birthday paradox puts their probability at around 2^{−64+2·23.5}=2⁻¹⁷. We note that the communication complexity also reduces with the number of bits of the data types.

In our implementation, evaluation of E can take place either immediately after the data upload, or it can be run in a separate thread, waiting until more data comes to process it all in parallel. In any case, the application of E is done in parallelized manner, batching up the hashes that are waiting for being processed so far.

Computation of results.

When the upload has finished and the ciphertext ⟦c_i⟧ has been computed for each i∈{0,…,cnt−1}, then the final results are computed as in Algorithm 5. First of all, all computed ciphertexts are privately shuffled, and then declassified. Since they are encrypted with a block cipher whose key remains secretshared, it only reveals the “duplication pattern”, i.e. how many values occur there exactly n times. The private shuffle [26] in the shared3p protocol set is highly efficient: the amount of communication is linear in the size of the shuffled vector, and the number of communication rounds is constant. It also allows to easily apply the shuffle inverse, which is as efficient as the shuffle itself.

At this point, the servers could already mark the duplicates with boolean values 1 and the remaining values with 0. After the obtained vector is privately shuffled back, the servers can return the secret shared bits to the clients, so that the i-th clients learns the bits indicating the duplicity of its own records. The problem is that the servers need to notify all clients except the first one, but because of shuffling they do not know which entry belongs to the first client. They cannot declassify the client indices ⟦s_i⟧ either since it would partially undo the shuffling.

In order to determine which value belongs to the first client, the servers run Algorithm 6. This algorithm is applied to each set of shuffled entries that have identical values, to determine the minimum amongst them. The minimum should be labeled false since it is not considered a duplicate, and all other elements should be labeled true. The idea behind this recursive algorithm is the following. The inputs are split into pairs, and a comparison performed within each pair. The element that is larger is definitely not the minimum of the entire set, so we can immediately write b_i:=true. The indices of all elements that turned out to be smaller are stored into m_i, and the whole algorithm is now applied again to the elements indexed by m_i. The procedure is repeated until there are is only one element left, which is the minimum, so the algorithm returns false in one-element case.

Although Algorithm 6 works with private values, it reveals the ordering between certain elements of \(\vec {t}\). Because of the random shuffle, the ordering of this vector is completely random, so it does not disclose any information. No server learns from the output more than it would from a random permutation.

Computation on the client side

The clients perform pretty much in the same manner as in the first solution, according to Algorithm 1. The only technical difference is that the servers react to the upload immediately, so the client most likely does not interrupt the session with the servers, and stays connected until it receives the result.

Computation on the server side

Assume that the servers already store T encrypted hashes (zero or more), provided by the previous clients. These T hashes have been stored as public values z₁,…,z_T, where z_i is equal to the first 64 bits of E(K,v_i). Duplicates have already been removed among z₁,…,z_T. Assuming that duplicates never end up among z₁,…,z_T, since ⟦K⟧ remains private, z₁,…,z_T are computationally indistinguishable from T uniformly randomly distributed values, so it is safe to make them public.

The 64-bit values z₁,…,z_T are kept in 2^Bbuckets, numbered from 0 to 2^B−1. In our implementation, B=16. Each bucket is just a vector of values. Each z_i is assigned to the bucket B_j, where j is equal to the first 16 bits of z_i.

Now suppose that a center has uploaded the hashes ⟦v₁⟧,…,⟦v_t⟧. The servers need to check whether these hashes has occurred before. The computation starts by encrypting each ⟦v_i⟧. Let \(\llbracket {z^{\prime }_{1}}\rrbracket,\ldots,\llbracket {z^{\prime }_{t}}\rrbracket \) be the results of encryption, computed by \(\llbracket {z^{\prime }_{i}}\rrbracket =\mathsf {{leftbits}_{64}}(E(\llbracket {K}\rrbracket,\llbracket {v_{i}}\rrbracket))\). We cannot immediately declassify them since it would leak which pairs of centers have intersecting sets of records. Instead, we should use privacy-preserving comparison.

We do not want to simply invoke the private comparison protocol for each z_i and \(\llbracket {z^{\prime }_{j}}\rrbracket \), because we consider their number to be too large. Indeed, as we are aiming to handle ca. 10 million records, this method would cause us to compare each pair of records, leading to ca. 5·10¹³ invocations of the comparison protocol. An ℓ-bit comparison requires slightly more network communication than a ℓ-bit multiplication [16], with the latter requiring each computation server to send and receive two ℓ-bit values [17]. If ℓ=64 and there are 5·10¹³ operations, then each server has to send out and receive at least 6·10¹⁵ bits, which on a 100 Mb/s network (specified in the conditions of the competition task) would take almost two years.

We reduce the number of comparisons in the following manner. Let ⟦z^′⟧ be one of the private values \(\llbracket {z^{\prime }_{1}}\rrbracket,\ldots,\llbracket {z^{\prime }_{t}}\rrbracket \); all t values are handled in parallel. The comparison of ⟦z^′⟧ with z₁,…,z_T is described in Algorithm 7. In this algorithm, we let N be the maximum size of a bucket. We denote the j-th element of the i-th bucket by B_i,j. We assume that each bucket has exactly N elements, adding special dummy elements if necessary.

The characteristic vector of an element \(x\in \mathbb {Z}_{M}\) is a vector \(\langle {b_{0},\ldots,b_{M-1}}\rangle \in \mathbb {Z}_{2}^{M}\), where b_x=1 and all other elements are equal to 0. The shared3p protocol set has a simple and efficient protocol for computing characteristic vectors, described in [27]. The protocol turns a private value into a private characteristic vector. The characteristic vector of leftbits₁₆(⟦z^′⟧) marks the index of the bucket to which ⟦z^′⟧ belongs, and the expression \(\bigoplus _{i=0}^{2^{B}-1}\llbracket {b_{i}}\rrbracket \cdot \mathbf {B}_{i,j}\) returns exactly the j-th element of that bucket, which we denote ⟦y_j⟧. This way, the values ⟦y₁⟧,…,⟦y_N⟧ are the privately represented content of the bucket, into which z^′ would belong. The private comparison \(\llbracket {z^{\prime }}\rrbracket \stackrel {?}{=}\llbracket {y_{j}}\rrbracket \) is performed for all j, thus comparing ⟦z^′⟧ against each element that belongs to the i-th bucket. Finally, \(\llbracket {b}\rrbracket =\bigvee _{j=1}^{n}\llbracket {c_{j}}\rrbracket \) is the private OR of all comparisons, which tells whether there had been at least one match. The private bits ⟦b₁⟧,…,⟦b_t⟧ resulting from applying Algorithm 7 to all \(\llbracket {z^{\prime }_{1}}\rrbracket,\ldots,\llbracket {z^{\prime }_{t}}\rrbracket \) are returned to the client.

In our implementation, ⟦z^′⟧ is shared over \(\mathbb {Z}_{2}^{64}\), hence taking the first 16 bits of it is a local operation, resulting in a value in \(\mathbb {Z}_{2}^{16}\). The characteristic vector protocol in [27] is easily adaptable to compute the characteristic vectors of elements of \(\mathbb {Z}_{2}^{B}\), and its result is a vector over \(\mathbb {Z}_{2}\) with length 2^B. The computation of the characteristic vector requires communication of two elements of \(\mathbb {Z}_{2}^{B}\) and one element of \(\mathbb {Z}_{2}^{2^{B}}\) (in total, not per party).

The computation of ⟦y_j⟧ in Algorithm 7 is again a local operation. The computations of ⟦c_j⟧ and their disjunction are straightforward using the protocols available in the shared3p set of Sharemind.

Speeding up local computations

In Alg. 7, the computation of ⟦y_j⟧ is local. Nevertheless, in practice it takes a major part of the entire effort of the protocol, taking up most of the time in it. If we knew something about the size of z^′, we could compute \(\bigoplus \) not over all buckets, but only over a subset of them, covering only the range into which z^′ is guaranteed to fall, with a negligible error.

A bucket is defined by the most significant bits of elements that it contains. If we take any two buckets, then all elements in one of them will be strictly smaller than all elements in the other one. If we sort \(\llbracket {z^{\prime }_{1}}\rrbracket,\ldots,\llbracket {z^{\prime }_{t}}\rrbracket \) in ascending order (Sharemind has efficient protocols for sorting private values [28]), we know that the first elements more likely belong to the buckets with “smaller” bits, and the last elements more likely belong to buckets with “larger” bits. We can estimate these probabilities more precisely.

As the key ⟦K⟧ is secret, and the hashes ⟦v₁⟧,…,⟦v_t⟧ are all different, the values \(\llbracket {z^{\prime }_{1}}\rrbracket,\ldots,\llbracket {z^{\prime }_{t}}\rrbracket \) can be treated as mutually independent, uniformly random elements of \(\mathbb {Z}_{2}^{64}\). After sorting them, their likely ranges can be derived from the order statistics as follows.

Let \(\mathcal {P}\) be a discrete probability distribution over values x₁,x₂,…, such that the probability mass of x_i is p_i. Let X₁,…,X_n be random variables sampled from \(\mathcal {P}\), and let \(X^{\prime }_{1},\ldots,X^{\prime }_{n}\) be obtained after sorting X₁,…,X_n in ascending order. We have \(\text {Pr}[X^{\prime }_{j} \leq x_{i}] = \sum _{k=j}^{n}{{n}\choose {k}}P_{i}^{k} \cdot (1 - P_{i})^{n-k}\), where \(P_{i} = Pr[X_{i} \leq x_{i}] = \sum _{k=1}^{i} p_{k}\). This quantity comes from summing up probabilities of all possible combinations, where at least j of n variables are smaller than x_i. For a fixed j, this expression is actually the cumulative density function of a binomial distribution B(n,P_j).

By default, we use ε=2⁻⁴⁰ as the probability of error. As the total number of hashes is expected to be around 1000·10000≈2^23.5 and we have two bounds to try for each hash, the probability of making a bounds check error during the whole run is not more than 2·2^−40+23.5=2^−15.5, which we consider acceptable, and which is also similar to errors due to the collisions in the first 64 bits of AES output.

The usefulness of these bounds increases together with t. If t=100 (and ε=2⁻⁴⁰), then we gain little, as the ranges [m_i,M_i] still cover around half of the whole range. If t=10000, then the sorted values can be much more tightly positioned — the ranges [m_i,M_i] are less than 1/10 of the whole range.

Alternative comparison

The communication costs of Algorithm 7 may be further reduced, ultimately turning them into a constant (assuming that B is constant), albeit with a further increase in the costs of local computation.

Consider the bucket B_i with elements B_i,1,…,B_i,N. The value z^′ is an element B_i iff it is a root of the polynomial \(\mathbf {P}_{i}(x)=\prod _{j=1}^{N}(x-\mathbf {B}_{i,j})\). The polynomial is considered over the field \(\mathbb {F}_{2^{64}}\). The elements of this field are 64-bit strings and their addition is bitwise exclusive or. Hence, an additive sharing over \(\mathbb {F}_{2^{64}}\) is at the same time also sharing over \(\mathbb {Z}_{2}^{64}\) and vice versa.

Let P_i,0,…,P_i,N be the coefficients of the polynomial P_i. It does not make sense to compute P_i(⟦z^′⟧) in a straightforward way, because this would involve N−1 private multiplications for each bucket. A better way is given in Algorithm 9.

In Algorithm 9, ⟦d_i⟧ is computed as a scalar product of the private vector of the powers of z^′ with the public vector of the coefficients of P_i. The powers of ⟦z^′⟧ have to be computed only once for all buckets. These powers are computed with the help of the usual multiplication protocol of Sharemind, but working over \(\mathbb {F}_{2^{64}}\). The local operations in this protocol are multiplications in \(\mathbb {F}_{2^{64}}\), which are relatively more expensive than ordinary bitwise operations. In our implementation we use the NTL library [29] for binary field operations. The computation of all ⟦d_i⟧ involves many multiplications in this field, so even though the computation does not require any communication between the parties, it is quite heavy on the local side.

As the computation of ⟦z^′2⟧,…,⟦z^′N⟧ is done over a field, we can push its heaviest part to the precomputation phase, leaving just a single private multiplication to be done during runtime. The method is described in ([30], Algorithm 1). The precomputation consists of generating a random invertible element \(\llbracket {r}\rrbracket \in \mathbb {F}_{2^{64}}^{*}\) together with its inverse ⟦r⁻¹⟧ and computing ⟦r²⟧,…,⟦r^N⟧. During the runtime, one computes ⟦z^′⟧·⟦r⁻¹⟧ and declassifies it. For an exponent k, the private value ⟦z^′k⟧ is then found as ⟦z^′k⟧=(z^′·r⁻¹)^k·⟦r^k⟧, which can be computed locally, without interaction.

In binary fields, including \(\mathbb {F}_{2^{64}}\), squaring of additively shared values does not require communication between the servers. This can be used to speed up precomputations. In ([30], Algorithm 4) it is shown how to reduce the communication cost of computing ⟦r²⟧,…,⟦r^N⟧ to that of approximately \(\sqrt {N}\) multiplications.

The polynomial-based comparison method is also amenable to the order statistic related speedup described in the previous section. Both comparison methods have been implemented in our second solution.