Sometimes in order to add one has to take the difference.
Yes, that's true provided the difference is symmetric. Several operations are customarily defined for general sets - union, intersection, difference:

1. Union: x∈A∪B iff either x∈A or x∈B
2. Intersection: x∈A∩B iff x∈A and x∈B
3. Difference: x∈A-B iff x∈A and x∉B

Adding elements of one set to another, only the union is intuitively suitable to be considered as the set addition. The empty set Ø plays the role of zero. Indeed, for every set A, A∪Ø = Ø∪A = A. The union is clearly associative. However, it's impossible to find a set (-A) such that A∪(-A) = Ø if A itself is not empty. This is because the union of two sets is a superset of each operand. There is one additional set operation that is worth paying attention to:

1. Symmetric Difference: x∈A^B iff either x∈A or x∈B but x∉A∩B

There are several ways to define this operation:

1. A^B = (A∪B) - (A∩B)
2. A^B = (A - B)∪(B - A)

The latter is obviously suggestive of the name's origin. A nice feature of this operation is that, for any set A, A^A = Ø so that in an algraic sense A = -A, and if symmetric difference satisfies the rest of the conditions, it may be legitimately called a "set addition". Clearly A^Ø = Ø^A = A. Also, the operation is commutative by definition. It's a good exercise to check that it's also associative.

Symmetric difference is reminiscent of the XOR operation defined for Boolean Algebras. So that the latter may be considered as addition defined for Boolean Algebras.  