Skip to content

Add legal hold membership to device reporting #192

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 23 commits into from
Mar 1, 2021
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
b285a68
Legal Hold work to meet Issue 176
maddie-vargo Dec 28, 2020
ed72879
Fix to Changelog
maddie-vargo Dec 28, 2020
ea4381a
Minor fix to CHANGELOG
maddie-vargo Dec 29, 2020
a0ebe57
Added to legal hold user guide
maddie-vargo Dec 29, 2020
077dea1
Adjusting build parameters to bypass 3.5 for this PR
maddie-vargo Dec 29, 2020
03a3814
Merge branch 'master' into iss176
maddie-vargo Dec 29, 2020
f174992
Fix low hanging fruit for initial PR review
maddie-vargo Jan 4, 2021
205d559
Move development from legal-hold command to devices command, add new …
maddie-vargo Jan 29, 2021
02d6530
remove whitespaces that are coming through as edits
maddie-vargo Feb 2, 2021
e5784cf
fix changes identfied by tox style run
maddie-vargo Feb 2, 2021
9bfcd42
remove duplication in setup.py - file should have no edits
maddie-vargo Feb 2, 2021
2e303de
remove duplication in setup.py - file should have no edits
maddie-vargo Feb 2, 2021
94fe3b9
refactor membership function to use generator and remove NaNs from ou…
maddie-vargo Feb 11, 2021
0b79911
fix tox style run issue
maddie-vargo Feb 11, 2021
1bd1e2f
Fix tox style run x2
maddie-vargo Feb 11, 2021
e4725c7
flipping back to using NaN, awaiting PR #245
maddie-vargo Feb 16, 2021
237ea31
Adding --include-total-storage option, which calculates total number …
maddie-vargo Feb 18, 2021
7212fbf
Remove V2 archives from storage calcuation; rename columns
maddie-vargo Feb 22, 2021
2a917bd
fix small change to the incldue/excluded archive types
maddie-vargo Feb 23, 2021
2d1db8c
reword
maddie-vargo Feb 25, 2021
9a0afcb
conflict reconciliation in changelog, part I
maddie-vargo Feb 26, 2021
a3dd28f
conflict reconciliation in changelog, part II (repulled from upstream…
maddie-vargo Feb 26, 2021
9677f23
fix style run
maddie-vargo Feb 26, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python: [3.5, 3.6, 3.7, 3.8]
python: [3.6, 3.7, 3.8]

steps:
- uses: actions/checkout@v2
Expand Down
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ how a consumer would use the library (e.g. adding unit tests, updating documenta
- `search` to search for audit-logs.
- `send-to` to send audit-logs to server.

- `code42 legal-hold show` option:
Copy link
Contributor

@antazoey antazoey Jan 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do have a new devices list command that is merged by not yet released.

Would it make more sense to have a command code42 devices list --matter-id <UID> instead?

Just want to make sure all the options are considered, I don't really understand the use-case.

Copy link
Contributor Author

@maddie-vargo maddie-vargo Jan 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's an interesting point. I chose to add it to legal-hold show because that command already showed details on a legal hold matter object, including users on legal hold and (optional) preservation policy details. I thought showing devices on legal hold would be a logical fit here.

From what I gather, the devices command renders mostly the Computer api resource. Unfortunately, there is no flag in the Computer api response that states whether a device is on legal hold. In order to pull legal hold devices, you first need to pull legal hold membership to get a list of users and then find the devices associated with those users. It would not be impossible to move this into the devices command, but we would still have to run some sort of legal hold py42 resource to determine the scope of devices.

Pinging @ceciliastevens for input as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or a code42 legal-hold list-devices [MATTER-ID]. Trying to find a way to afford output format options (At least table and csv).

Copy link
Contributor Author

@maddie-vargo maddie-vargo Jan 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, if we can find a way to structure this to add a format option, that would be helpful to most customers.

Currently, the legal-hold show command outputs data in the following formats:

  • Matter info - table
  • Users - columns
  • (optional) Preservation policy - JSON

What do you think about removing users from the show command and creating code42 legal-hold list-users [Matter ID] with an --include-devices optiion? I think it makes sense to keep users and devices closely tied together, rather than in separate commands. From a code perspective, to get devices, you'll have to get users first, anyway.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that! @timabrmsn What are you thoughts on this? I believe you were the one who implemented the show command.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually think it makes more sense to add legal hold status as an option to the devices list command, since that would be a useful option for people who are creating a report of all their devices to know which ones are on legal hold quickly when taking inventory of other things. It could just add a legalHoldMatterId column to the output.

And keeping users under legal-hold show still makes sense to me, since it's a high level overview of the matter, of which membership is a key detail. Otherwise the show command just becomes a single row of the list command with the option to also show preservation policy.

devices list already has --include-usernames which makes a separate call to api/Users and then merges the DataFrames on userId, so adding legal hold status would be easy enough to add using the same logic.

The storage total report is an interesting idea, but might be better suited to include in devices list report also, so it can be used more generically and not tied directly to only legal hold. I'm thinking something like --include-storage-totals on devices list that just adds a column per device for the sum of all that device's archives. Then if someone wanted to get a report broken down by matter Id, they could just filter by matter_id and sum the total in the csv report.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, devices it is. The device pull actually includes userUid without the username option, so it's even easier to merge in the legal hold data.

There are two main things that we want this to accomplish:

  • Limit the device list to devices on a specific legal hold matter
    devices --matter-uid - option to limit device search base on a specific legal hold matter, adds in columns such as legalHoldStatus, legalHoldMatterId, legalHoldMatterName

  • Add in legal hold membership columns for whatever device query you're running.
    devices -include-legal-hold-membership - option to option to include legal hold membership on to the devices table, adds same columns as above but doesn't limit the population

This feels a little redundant, so any feedback is appreciated.


As for storage-totals: is a sum of all device's archives appropriate? (this is how I wrote my first draft). However, the current legal hold script does some logic to determine which destination is the most recent and complete, and only uses that archive size as the total storage.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that since filtering by matter-id outside of the CLI is simple enough (i.e. grep/Select-String or directly in excel), can we leave that to the end-user? I'd like to not have our option list expand too much. And I have some ideas for more generic column/row filtering logic that could work on any cmd that outputs a dataframe. Something like `--filter 'matter-id == 42' that would convert to a pandas expression under the hood. Although I haven't started trying to implement that so I'm not sure how feasible it is...

Also, because a device can be part of multiple legal holds, we need to think of how to display that. Can we just have a single column that has a list of matter-uids the device is part of, maybe separated by semi-colons or something?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, for storage totals I think adding a usedStorage column that sums the total combined archive size for a device would work, maybe we could also add an archiveCount column that gets added with that option?

Trying to keep what features we add as generic and multi-use-case friendly as possible.

- `--include-devices` to print list of devices associated with legal hold along with total storage by organization.

### Changed

- `profile_name` argument is now required for `code42 profile delete`, as it was meant to be.
Expand Down
4 changes: 4 additions & 0 deletions docs/userguides/legalhold.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,4 +93,8 @@ To view all custodians (including inactive) for a legal hold matter, enter

`code42 legal-hold show <matterID> --include-inactive`

To view all devices associated with selected custodians for a legal hold matter, enter

`code42 legal-hold show <matterID> --include-devices`

Learn more about the [Legal Hold](../commands/legalhold.md) commands.
4 changes: 2 additions & 2 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,13 @@
package_dir={"": "src"},
include_package_data=True,
zip_safe=False,
python_requires=">3, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, <4",
python_requires=">3, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, <4",
install_requires=[
"click>=7.1.1",
"colorama>=0.4.3",
"c42eventextractor==0.4.0",
"keyring==18.0.1",
"pandas>=1.1.3",
"keyrings.alt==3.2.0",
"py42>=1.10",
],
Expand All @@ -53,7 +54,6 @@
"License :: OSI Approved :: MIT License",
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.5",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
Expand Down
87 changes: 78 additions & 9 deletions src/code42cli/cmds/legal_hold.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

import click
from click import echo
from pandas import DataFrame

from code42cli.bulk import generate_template_cmd_factory
from code42cli.bulk import run_bulk_process
Expand Down Expand Up @@ -92,8 +93,19 @@ def _list(state, format=None):
is_flag=True,
help="View details of the preservation policy associated with the legal hold matter.",
)
@click.option(
"--include-devices",
is_flag=True,
help="View devices and storage associated with legal hold custodians.",
)
@sdk_options()
def show(state, matter_id, include_inactive=False, include_policy=False):
def show(
state,
matter_id,
include_inactive=False,
include_policy=False,
include_devices=False,
):
"""Display details of a given legal hold matter."""
matter = _check_matter_is_accessible(state.sdk, matter_id)
matter["creator_username"] = matter["creator"]["username"]
Expand All @@ -105,19 +117,32 @@ def show(state, matter_id, include_inactive=False, include_policy=False):
memberships = _get_legal_hold_memberships_for_matter(
state.sdk, matter_id, active=active
)
active_usernames = [
member["user"]["username"] for member in memberships if member["active"]
]
inactive_usernames = [
member["user"]["username"] for member in memberships if not member["active"]
]

formatter = OutputFormatter(OutputFormat.TABLE, _MATTER_KEYS_MAP)
formatter.echo_formatted_list([matter])
_print_matter_members(active_usernames, member_type="active")

users = [
[member["active"], member["user"]["userUid"], member["user"]["username"]]
for member in memberships
]

usernames = [user[2] for user in users if user[0] is True]
_print_matter_members(usernames, member_type="active")
if include_inactive:
_print_matter_members(inactive_usernames, member_type="inactive")
usernames = [user[2] for user in users if user[0] is not True]
_print_matter_members(usernames, member_type="inactive")

if include_devices:
user_dataframe = _build_user_dataframe(users)
devices_dataframe = _merge_matter_members_with_devices(
state.sdk, user_dataframe
)
if len(devices_dataframe.index) > 0:
echo("\nMatter Members and Devices:\n")
click.echo(devices_dataframe.to_csv())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We typically use table format for show commands.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

List commands, however, are known to support various output formats, inc. CSV.... My first comment on this PR suggests that this would make sense as a code42 devices list --matter-id <UID> command, which then you can tack on -f CSV or -f JSON etc.... Would that be better?

echo(_print_storage_by_org(devices_dataframe))
else:
echo("\nNo devices associated with matter.\n")

if include_policy:
_get_and_print_preservation_policy(state.sdk, matter["holdPolicyUid"])
Expand Down Expand Up @@ -238,6 +263,50 @@ def _print_matter_members(username_list, member_type="active"):
echo("No {} matter members.\n".format(member_type))


def _merge_matter_members_with_devices(sdk, user_dataframe):
devices_generator = sdk.devices.get_all(active="true", include_backup_usage=True)
device_list = _get_total_archive_bytes_per_device(devices_generator)
devices_dataframe = DataFrame.from_records(
device_list,
columns=[
"userUid",
"guid",
"name",
"osHostname",
"status",
"alertStates",
"orgId",
"lastConnected",
"version",
"archiveBytes",
],
)
return user_dataframe.merge(
devices_dataframe, how="inner", on="userUid"
).reset_index(drop=True)


def _build_user_dataframe(users):
user_dataframe = DataFrame.from_records(
users, columns=["activeMembership", "userUid", "username"]
)
return user_dataframe


def _get_total_archive_bytes_per_device(devices_generator):
device_list = [device for page in devices_generator for device in page["computers"]]
for i in device_list:
archive_bytes = [archive["archiveBytes"] for archive in i["backupUsage"]]
i["archiveBytes"] = sum(archive_bytes)
return device_list


def _print_storage_by_org(devices_dataframe):
echo("\nLegal Hold Storage by Org\n")
devices_dataframe = devices_dataframe.filter(["orgId", "archiveBytes"])
return devices_dataframe.groupby("orgId").sum()


@lru_cache(maxsize=None)
def _check_matter_is_accessible(sdk, matter_id):
return sdk.legalhold.get_matter_by_uid(matter_id)
Loading