Skip to content

CSHARP-4255: Automatically create Queryable Encryption keys. #961

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Dec 14, 2022
Merged

CSHARP-4255: Automatically create Queryable Encryption keys. #961

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
1f05fe2
CSHARP-4255: Automatically create Queryable Encryption keys.
DmitryLukyanov Nov 21, 2022
71e171e
Test fix.
DmitryLukyanov Nov 22, 2022
9318348
Code review,
DmitryLukyanov Nov 25, 2022
2b8e02a
Code review
DmitryLukyanov Nov 25, 2022
8b8705c
Fix typo.
DmitryLukyanov Dec 7, 2022
c0b7f6a
Fix rewrapProseTest
DmitryLukyanov Dec 7, 2022
c0e4ab1
Removed return type.
DmitryLukyanov Dec 7, 2022
6013fc2
Add positive scenario.
DmitryLukyanov Dec 13, 2022
4e6d6b9
Code review.
DmitryLukyanov Dec 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions src/MongoDB.Driver.Core/Core/Encryption/EncryptedCollectionHelper.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,40 @@ public static BsonDocument GetEffectiveEncryptedFields(CollectionNamespace colle
}
}

public static IEnumerable<BsonDocument> IterateEmptyKeyIds(CollectionNamespace collectionNamespace, BsonDocument encryptedFields)
{
if (!EncryptedCollectionHelper.TryGetEffectiveEncryptedFields(collectionNamespace, encryptedFields, encryptedFieldsMap: null, out var storedEncryptedFields))
{
throw new InvalidOperationException("There are no encrypted fields defined for the collection.");
}

if (storedEncryptedFields.TryGetValue("fields", out var fields) && fields is BsonArray fieldsArray)
{
foreach (var field in fieldsArray)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fieldsArray.OfType<BsonDocument>() ?
or even

fieldsArray
    .OfType< BsonDocument>()
    .Where(b => b.TryGetElement("keyId", out var keyId) && keyId.Value == BsonNull.Value)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, I like first one, second one probably is a bit harder to read, done

{
if (field is BsonDocument fieldDocument)
{
if (fieldDocument.TryGetElement("keyId", out var keyId) && keyId.Value == BsonNull.Value)
{
yield return fieldDocument;
}
}
else
{
// If `F` is not a document element, skip it.
continue;
}
}
}

yield break;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not needed?

}

public static void ModifyEndryptedFields(BsonDocument fieldDocument, Guid dataKey)
{
fieldDocument["keyId"] = new BsonBinaryData(dataKey, GuidRepresentation.Standard);
}

public enum HelperCollectionForEncryption
{
Esc,
Expand Down
58 changes: 58 additions & 0 deletions src/MongoDB.Driver/Encryption/ClientEncryption.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,64 @@ public BsonDocument AddAlternateKeyName(Guid id, string alternateKeyName, Cancel
public Task<BsonDocument> AddAlternateKeyNameAsync(Guid id, string alternateKeyName, CancellationToken cancellationToken = default) =>
_libMongoCryptController.AddAlternateKeyNameAsync(id, alternateKeyName, cancellationToken);

/// <summary>
/// Create encrypted collection.
/// </summary>
/// <param name="collectionNamespace">The collection namespace.</param>
/// <param name="createCollectionOptions">The create collection options.</param>
/// <param name="kmsProvider">The kms provider.</param>
/// <param name="dataKeyOptions">The datakey options.</param>
/// <param name="cancellationToken">The cancellation token.</param>
public (IMongoCollection<TCollection> Collection, BsonDocument EncryptedFields) CreateEncryptedCollection<TCollection>(CollectionNamespace collectionNamespace, CreateCollectionOptions createCollectionOptions, string kmsProvider, DataKeyOptions dataKeyOptions, CancellationToken cancellationToken = default)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to discuss whether we can add a different return type than this, so might be not the final form

Copy link
Contributor Author

@DmitryLukyanov DmitryLukyanov Dec 7, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've discussed this question with spec author and he's confirmed that we can define public API in the consistent way with previous driver's API methods. So, I consider 2 changes:

  1. Do not return created collection from this helper method. Instead do the same logic as we do with a regular CreateCollection method ie return nothing and request retrieving collection via GetCollection.
  2. Instead returning EncryptedFields, we can simply modify input EncryptedFields in the provided options.

I'm not too confident about these changes (as well as about the initial requirement), but I would probably want to avoid returning tuple in public API, thoughts?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think Tuples are not recommended in public APIs.

Agreed that there is no need to return the IMongoCollection<TDocument> .

Do we need to return anything?

Also, shouldn't the type parameter be TDocument (not TCollection). But if we don't return the IMongoCollection<TDocument> I don't think this type parameter is needed any more anyway.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to return anything?

yeah, I also think it can be simply void method

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed

{
var effectiveEncryptedFields = createCollectionOptions?.EncryptedFields?.DeepClone()?.AsBsonDocument;

foreach (var fieldDocument in EncryptedCollectionHelper.IterateEmptyKeyIds(collectionNamespace, effectiveEncryptedFields))
{
var dataKey = CreateDataKey(kmsProvider, dataKeyOptions, cancellationToken);
EncryptedCollectionHelper.ModifyEndryptedFields(fieldDocument, dataKey);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like ModifyEndryptedFields is spelled wrong.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

true

}

var database = _libMongoCryptController.KeyVaultClient.GetDatabase(collectionNamespace.DatabaseNamespace.DatabaseName);

createCollectionOptions.EncryptedFields = effectiveEncryptedFields;

database.CreateCollection(collectionNamespace.CollectionName, createCollectionOptions, cancellationToken);

var collection = database.GetCollection<TCollection>(collectionNamespace.CollectionName);

return (collection, effectiveEncryptedFields);
}

/// <summary>
/// Create encrypted collection.
/// </summary>
/// <param name="collectionNamespace">The collection namespace.</param>
/// <param name="createCollectionOptions">The create collection options.</param>
/// <param name="kmsProvider">The kms provider.</param>
/// <param name="dataKeyOptions">The datakey options.</param>
/// <param name="cancellationToken">The cancellation token.</param>
public async Task<(IMongoCollection<TCollection> Collection, BsonDocument EncryptedFields)> CreateEncryptedCollectionAsync<TCollection>(CollectionNamespace collectionNamespace, CreateCollectionOptions createCollectionOptions, string kmsProvider, DataKeyOptions dataKeyOptions, CancellationToken cancellationToken = default)
{
var effectiveEncryptedFields = createCollectionOptions?.EncryptedFields?.DeepClone()?.AsBsonDocument;

foreach (var fieldDocument in EncryptedCollectionHelper.IterateEmptyKeyIds(collectionNamespace, effectiveEncryptedFields))
{
var dataKey = await CreateDataKeyAsync(kmsProvider, dataKeyOptions, cancellationToken).ConfigureAwait(false);
EncryptedCollectionHelper.ModifyEndryptedFields(fieldDocument, dataKey);
}

var database = _libMongoCryptController.KeyVaultClient.GetDatabase(collectionNamespace.DatabaseNamespace.DatabaseName);

createCollectionOptions.EncryptedFields = effectiveEncryptedFields;

await database.CreateCollectionAsync(collectionNamespace.CollectionName, createCollectionOptions, cancellationToken).ConfigureAwait(false);

var collection = database.GetCollection<TCollection>(collectionNamespace.CollectionName);

return (collection, effectiveEncryptedFields);
}

/// <summary>
/// An alias function equivalent to createKey.
/// </summary>
Expand Down
3 changes: 3 additions & 0 deletions src/MongoDB.Driver/Encryption/LibMongoCryptControllerBase.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,9 @@ protected LibMongoCryptControllerBase(
_tlsOptions = Ensure.IsNotNull(encryptionOptions.TlsOptions, nameof(encryptionOptions.TlsOptions));
}

// public proeprties
public IMongoClient KeyVaultClient => _keyVaultClient;

// protected methods
protected void FeedResult(CryptContext context, BsonDocument document)
{
Expand Down
81 changes: 77 additions & 4 deletions ...ver.Tests/Specifications/client-side-encryption/prose-tests/ClientEncryptionProseTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@
using MongoDB.Driver.Core.Authentication.External;
using MongoDB.Driver.Core.Bindings;
using MongoDB.Driver.Core.Clusters;
using MongoDB.Driver.Core.Configuration;
using MongoDB.Driver.Core.Events;
using MongoDB.Driver.Core.Misc;
using MongoDB.Driver.Core.Operations;
Expand Down Expand Up @@ -84,6 +83,69 @@ public ClientEncryptionProseTests(ITestOutputHelper testOutputHelper)
}

// public methods
[SkippableTheory]
[ParameterAttributeData]
public void AutomaticDataEncryptionKeys(
[Range(1, 3)] int testCase,
[Values(false, true)] bool async)
{
RequireServer.Check().Supports(Feature.Csfle2).ClusterTypes(ClusterType.ReplicaSet, ClusterType.Sharded, ClusterType.LoadBalanced);

var kmsProvider = "local";
using (var client = ConfigureClient())
using (var clientEncryption = ConfigureClientEncryption(client, kmsProviderFilter: kmsProvider))
{
var encryptedFields = BsonDocument.Parse($@"
{{
fields:
[
{{
path: ""ssn"",
bsonType: ""string"",
keyId: null
}}
]
}}");

DropCollection(__collCollectionNamespace, encryptedFields);

RunTestCase(testCase);

void RunTestCase(int testCase)
{
switch (testCase)
{
case 1: // Case 1: Simple Creation and Validation
{
var collection = CreateEncryptedCollection(client, clientEncryption, __collCollectionNamespace, encryptedFields, kmsProvider, async).Collection;

var exception = Record.Exception(() => Insert(collection, async, new BsonDocument("ssn", "123-45-6789")));
exception.Should().BeOfType<MongoBulkWriteException<BsonDocument>>().Which.Message.Should().Contain("Document failed validation");
}
break;
case 2: // Case 2: Missing ``encryptedFields``
{
var exception = Record.Exception(() => CreateEncryptedCollection(client, clientEncryption, __collCollectionNamespace, encryptedFields: null, kmsProvider, async).Collection);

exception.Should().BeOfType<InvalidOperationException>().Which.Message.Should().Contain("There are no encrypted fields defined for the collection.") ;
}
break;
case 3: // Case 3: Invalid ``keyId``
{
var effectiveEncryptedFields = encryptedFields.DeepClone();
effectiveEncryptedFields["fields"].AsBsonArray[0].AsBsonDocument["keyId"] = true;
var collection = CreateEncryptedCollection(client, clientEncryption, __collCollectionNamespace, encryptedFields, kmsProvider, async).Collection;

var exception = Record.Exception(() => Insert(collection, async, new BsonDocument("ssn", "123-45-6789")));
exception.Should().BeOfType<MongoBulkWriteException<BsonDocument>>().Which.Message.Should().Contain("Document failed validation");
}
break;
default: throw new Exception($"Unexpected test case {testCase}.");
}
}
}
}

[SkippableTheory]
[ParameterAttributeData]
public void BsonSizeLimitAndBatchSizeSplittingTest(
Expand Down Expand Up @@ -1025,6 +1087,7 @@ void RunTestCase(IMongoCollection<BsonDocument> decryptionEventsCollection, int
reply["cursor"]["firstBatch"].AsBsonArray.Single()["encrypted"].AsBsonBinaryData.SubType.Should().Be(BsonBinarySubType.Encrypted);
}
break;
default: throw new Exception($"Unexpected test case {testCase}.");
}
}

Expand Down Expand Up @@ -1873,7 +1936,7 @@ public void ViewAreProhibitedTest([Values(false, true)] bool async)
using (var client = ConfigureClient(false))
using (var clientEncrypted = ConfigureClientEncrypted(kmsProviderFilter: "local"))
{
DropView(viewName);
DropCollection(viewName);
client
.GetDatabase(viewName.DatabaseNamespace.DatabaseName)
.CreateView(
Expand Down Expand Up @@ -2201,6 +2264,16 @@ private void CreateCollection(IMongoClient client, CollectionNamespace collectio
});
}

private (IMongoCollection<BsonDocument> Collection, BsonDocument EncryptedFields) CreateEncryptedCollection(IMongoClient client, ClientEncryption clientEncryption, CollectionNamespace collectionNamespace, BsonDocument encryptedFields, string kmsProvider, bool async)
{
var createCollectionOptions = new CreateCollectionOptions { EncryptedFields = encryptedFields };
var datakeyOptions = CreateDataKeyOptions(kmsProvider);

return async
? clientEncryption.CreateEncryptedCollectionAsync<BsonDocument>(collectionNamespace, createCollectionOptions, kmsProvider, datakeyOptions, cancellationToken: default).GetAwaiter().GetResult()
: clientEncryption.CreateEncryptedCollection<BsonDocument>(collectionNamespace, createCollectionOptions, kmsProvider, datakeyOptions, cancellationToken: default);
}

private Guid CreateDataKey(
ClientEncryption clientEncryption,
string kmsProvider,
Expand Down Expand Up @@ -2351,9 +2424,9 @@ private MongoClientSettings CreateMongoClientSettings(
return mongoClientSettings;
}

private void DropView(CollectionNamespace viewNamespace)
private void DropCollection(CollectionNamespace viewNamespace, BsonDocument encryptedFields = null)
{
var operation = new DropCollectionOperation(viewNamespace, CoreTestConfiguration.MessageEncoderSettings);
var operation = DropCollectionOperation.CreateEncryptedDropCollectionOperationIfConfigured(viewNamespace, encryptedFields, CoreTestConfiguration.MessageEncoderSettings, configureDropCollectionConfigurator: null);
using (var session = CoreTestConfiguration.StartSession(_cluster))
using (var binding = new WritableServerBinding(_cluster, session.Fork()))
using (var bindingHandle = new ReadWriteBindingHandle(binding))
Expand Down