Skip to content
This repository was archived by the owner on Mar 13, 2025. It is now read-only.

Commit 1001b49

Browse files
callmekatootiecallmekatootie
committed
Initial commit
0 parents  commit 1001b49

23 files changed

+6986
-0
lines changed

‎.gitignore

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
node_modules
2+
.nyc_output
3+
coverage
4+
5+
.DS_Store

‎README.md

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
# UBahn - Elasticsearch Data Processor
2+
3+
## Dependencies
4+
5+
- Nodejs(v12+)
6+
- ElasticSearch
7+
- Kafka
8+
9+
## Configuration
10+
11+
Configuration for the ubahn es processor is at `config/default.js`.
12+
The following parameters can be set in config files or in env variables:
13+
14+
- LOG_LEVEL: the log level; default value: 'debug'
15+
- KAFKA_URL: comma separated Kafka hosts; default value: 'localhost:9092'
16+
- KAFKA_CLIENT_CERT: Kafka connection certificate, optional; default value is undefined;
17+
if not provided, then SSL connection is not used, direct insecure connection is used;
18+
if provided, it can be either path to certificate file or certificate content
19+
- KAFKA_CLIENT_CERT_KEY: Kafka connection private key, optional; default value is undefined;
20+
if not provided, then SSL connection is not used, direct insecure connection is used;
21+
if provided, it can be either path to private key file or private key content
22+
- KAFKA_GROUP_ID: the Kafka group id, default value is 'ubahn-processor-es'
23+
- UBAHN_CREATE_TOPIC: the create ubahn entity Kafka message topic, default value is 'u-bahn.action.create'
24+
- UBAHN_UPDATE_TOPIC: the update ubahn entity Kafka message topic, default value is 'u-bahn.action.update'
25+
- UBAHN_DELETE_TOPIC: the delete ubahn entity Kafka message topic, default value is 'u-bahn.action.delete'
26+
- ES.HOST: Elasticsearch host, default value is 'localhost:9200'
27+
- ES.AWS_REGION: The Amazon region to use when using AWS Elasticsearch service, default value is 'us-east-1'
28+
- ES.API_VERSION: Elasticsearch API version, default value is '6.8'
29+
- ES.ACHIEVEMENT_PROVIDER_INDEX: Elasticsearch index name for achievement provider, default value is 'achievement_provider'
30+
- ES.ACHIEVEMENT_PROVIDER_TYPE: Elasticsearch index type for achievement provider, default value is '_doc'
31+
- ES.ATTRIBUTE_INDEX: Elasticsearch index name for attribute, default value is 'attribute'
32+
- ES.ATTRIBUTE_TYPE: Elasticsearch index type for attribute, default value is '_doc'
33+
- ES.ATTRIBUTE_GROUP_INDEX: Elasticsearch index name for attribute group, default value is 'attribute_group'
34+
- ES.ATTRIBUTE_GROUP_TYPE: Elasticsearch index type for attribute group, default value is '_doc'
35+
- ES.ORGANIZATION_INDEX: Elasticsearch index name for organization, default value is 'organization'
36+
- ES.ORGANIZATION_TYPE: Elasticsearch index type for organization, default value is '_doc'
37+
- ES.ROLE_INDEX: Elasticsearch index name for role, default value is 'role'
38+
- ES.ROLE_TYPE: Elasticsearch index type for role, default value is '_doc'
39+
- ES.SKILL_INDEX: Elasticsearch index name for skill, default value is 'skill'
40+
- ES.SKILL_TYPE: Elasticsearch index type for skill, default value is '_doc'
41+
- ES.SKILL_PROVIDER_INDEX: Elasticsearch index name for skill provider, default value is 'skill_provider'
42+
- ES.SKILL_PROVIDER_TYPE: Elasticsearch index type for skill provider, default value is '_doc'
43+
- ES.USER_INDEX: Elasticsearch index name for user, default value is 'user'
44+
- ES.USER_TYPE: Elasticsearch index type for user, default value is '_doc'
45+
- ES.USER_ACHIEVEMENT_PROPERTY_NAME: the user property name of achievement, default value is 'achievements',
46+
- ES.USER_EXTERNALPROFILE_PROPERTY_NAME: the user property name of externalProfile, default value is 'externalProfiles',
47+
- ES.USER_ATTRIBUTE_PROPERTY_NAME: the user property name of attribute, default value is 'attributes',
48+
- ES.USER_ROLE_PROPERTY_NAME: the user property name of role, default value is 'roles',
49+
- ES.USER_SKILL_PROPERTY_NAME: the user property name of skill, default value is 'skills'
50+
51+
There is a `/health` endpoint that checks for the health of the app. This sets up an expressjs server and listens on the environment variable `PORT`. It's not part of the configuration file and needs to be passed as an environment variable
52+
53+
Configuration for the tests is at `config/test.js`, only add such new configurations different from `config/default.js`
54+
55+
- WAIT_TIME: wait time used in test, default is 1500 or 1.5 second
56+
- ES.ACHIEVEMENT_PROVIDER_INDEX: Elasticsearch index name for achievement provider in testing environment
57+
- ES.ATTRIBUTE_INDEX: Elasticsearch index name for attribute in testing environment
58+
- ES.ATTRIBUTE_GROUP_INDEX: Elasticsearch index name for attribute group in testing environment
59+
- ES.ORGANIZATION_INDEX: Elasticsearch index name for organization in testing environment
60+
- ES.ROLE_INDEX: Elasticsearch index name for role in testing environment
61+
- ES.SKILL_INDEX: Elasticsearch index name for skill in testing environment
62+
- ES.SKILL_PROVIDER_INDEX: Elasticsearch index name for skill provider in testing environment
63+
- ES.USER_INDEX: Elasticsearch index name for user in testing environment
64+
65+
## Local Kafka and ElasticSearch setup
66+
67+
1. Navigate to the directory `docker-kafka-es`
68+
69+
2. Run the following command
70+
71+
```bash
72+
docker-compose up -d
73+
```
74+
75+
3. initialize Elasticsearch, create configured Elasticsearch index: `npm run init-es force`
76+
77+
## Local deployment
78+
79+
1. Make sure that Kafka and Elasticsearch is running as per instructions above.
80+
81+
2. From the project root directory, run the following command to install the dependencies
82+
83+
```bash
84+
npm install
85+
```
86+
87+
3. To run linters if required
88+
89+
```bash
90+
npm run lint
91+
```
92+
93+
To fix possible lint errors:
94+
95+
```bash
96+
npm run lint:fix
97+
```
98+
99+
4. Initialize Elasticsearch index
100+
101+
```bash
102+
npm run init-es
103+
```
104+
105+
To delete and re-create the index:
106+
107+
```bash
108+
npm run init-es force
109+
```
110+
111+
5. Start the processor and health check dropin
112+
113+
```bash
114+
npm start
115+
```
116+
117+
## Local Deployment with Docker
118+
119+
To run the UBahn ES Processor using docker, follow the below steps
120+
121+
1. Navigate to the directory `docker`
122+
123+
2. Rename the file `sample.api.env` to `api.env`
124+
125+
3. Set the required AWS credentials and ElasticSearch host in the file `api.env`
126+
127+
4. Once that is done, run the following command
128+
129+
```bash
130+
docker-compose up
131+
```
132+
133+
5. When you are running the application for the first time, It will take some time initially to download the image and install the dependencies
134+
135+
## Unit Tests and E2E Tests
136+
137+
- Run `npm run test` to execute unit tests.
138+
- Run `npm run test:cov` to execute unit tests and generate coverage report.
139+
- RUN `npm run e2e` to execute e2e tests.
140+
- RUN `npm run e2e:cov` to execute e2e tests and generate coverage report.
141+
142+
## Verification
143+
144+
see [VERIFICATION.md](VERIFICATION.md)

0 commit comments

Comments
 (0)