Skip to content

Commit 5761551

Browse files
author
sachin-maheshwari
authored
Merge pull request #106 from topcoder-platform/dev
General purpose processor service
2 parents 7235307 + f705b97 commit 5761551

20 files changed

+1527
-30
lines changed

‎.circleci/config.yml

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,12 @@ jobs:
5757
if [ -e dev-tc-notifications-deployvar.json ]; then sudo rm -vf dev-tc-notifications-deployvar.json; fi
5858
./buildenv.sh -e DEV -b dev-tc-notifications-consumers-deployvar
5959
source buildenvvar
60-
./master_deploy.sh -d ECS -e DEV -t latest -s dev-global-appvar,dev-tc-notifications-appvar -i tc-notifications
60+
./master_deploy.sh -d ECS -e DEV -t latest -s dev-global-appvar,dev-tc-notifications-appvar -i tc-notifications
61+
echo "Running Masterscript - deploy tc-notifications-general-processor service"
62+
if [ -e dev-tc-notifications-consumers-deployvar.json ]; then sudo rm -vf dev-tc-notifications-consumers-deployvar.json; fi
63+
./buildenv.sh -e DEV -b dev-tc-notifications-general-processor-deployvar
64+
source buildenvvar
65+
./master_deploy.sh -d ECS -e DEV -t latest -s dev-global-appvar,dev-tc-notifications-appvar -i tc-notifications
6166
6267
"build-prod":
6368
<<: *defaults
@@ -81,7 +86,12 @@ jobs:
8186
if [ -e prod-tc-notifications-deployvar.json ]; then sudo rm -vf prod-tc-notifications-deployvar.json; fi
8287
./buildenv.sh -e PROD -b prod-tc-notifications-consumers-deployvar
8388
source buildenvvar
84-
./master_deploy.sh -d ECS -e PROD -t latest -s prod-global-appvar,prod-tc-notifications-appvar -i tc-notifications
89+
./master_deploy.sh -d ECS -e PROD -t latest -s prod-global-appvar,prod-tc-notifications-appvar -i tc-notifications
90+
echo "Running Masterscript - prod deploy tc-notifications-general-processor service"
91+
if [ -e prod-tc-notifications-consumers-deployvar.json ]; then sudo rm -vf prod-tc-notifications-consumers-deployvar.json; fi
92+
./buildenv.sh -e PROD -b prod-tc-notifications-general-processor-deployvar
93+
source buildenvvar
94+
./master_deploy.sh -d ECS -e PROD -t latest -s prod-global-appvar,prod-tc-notifications-appvar -i tc-notifications
8595
8696
workflows:
8797
version: 2
@@ -92,7 +102,7 @@ workflows:
92102
context : org-global
93103
filters:
94104
branches:
95-
only: ['dev']
105+
only: [dev, 'feature/general-purpose-notifications-usage']
96106
- "build-prod":
97107
context : org-global
98108
filters:

‎Consumer-Verification.md

Lines changed: 221 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,221 @@
1+
# TOPCODER NOTIFICATIONS - CONSUMER VERIFICATION
2+
3+
## Local Kafka setup
4+
5+
- `http://kafka.apache.org/quickstart` contains details to setup and manage Kafka server,
6+
below provides details to setup Kafka server in Mac, Windows will use bat commands in bin/windows instead
7+
- download kafka at `https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.0/kafka_2.11-1.1.0.tgz`
8+
- extract out the downloaded tgz file
9+
- go to the extracted directory kafka_2.11-0.11.0.1
10+
- start ZooKeeper server:
11+
`bin/zookeeper-server-start.sh config/zookeeper.properties`
12+
- use another terminal, go to same directory, start the Kafka server:
13+
`bin/kafka-server-start.sh config/server.properties`
14+
- note that the zookeeper server is at localhost:2181, and Kafka server is at localhost:9092
15+
- use another terminal, go to same directory, create topics:
16+
```
17+
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic notifications.community.challenge.created
18+
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic notifications.community.challenge.phasewarning
19+
```
20+
21+
- verify that the topic is created:
22+
```
23+
bin/kafka-topics.sh --list --zookeeper localhost:2181
24+
```
25+
it should list out the created topics
26+
27+
- run producer and then write some message into the console to send to the `notifications.community.challenge.created` topic:
28+
```
29+
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic notifications.community.challenge.created
30+
```
31+
- In the console, write some message, one message per line:
32+
E.g.
33+
```
34+
{ "topic": "notifications.community.challenge.created", "originator": "tc-direct", "timestamp": "2018-02-16T00:00:00", "mime-type": "application/json", "payload": { "challengeId": 30054674, "challengeTitle": "test", "challengeUrl": "http://www.topcoder.com/123", "userId": 8547899, "initiatorUserId": 123, "skills": ["dotnet", "xcode"] } }
35+
```
36+
37+
- optionally, use another terminal, go to same directory, start a consumer to view the messages:
38+
```
39+
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic notifications.community.challenge.created --from-beginning
40+
```
41+
42+
43+
## Local deployment
44+
45+
- start local Kafka, start local PostgreSQL db, create an empty database `notification`
46+
- set some config params via env as below, use `set` instead of `export` for windows OS,
47+
instead, you may set them via `config/default.js`, modify the DATABASE_URL according to your setup db:
48+
```
49+
export LOG_LEVEL=debug
50+
export DATABASE_URL=postgres://postgres:123456@localhost:5432/notification
51+
export KAFKA_URL=localhost:9092
52+
export KAFKA_GROUP_ID=tc-notifications
53+
export ENV=test
54+
export [email protected]
55+
export [email protected]
56+
```
57+
58+
- to override TC API base URLs to use mock APIs, it is not needed if mock APIs are not used:
59+
```
60+
export TC_API_V3_BASE_URL=http://localhost:4000/v3
61+
export TC_API_V4_BASE_URL=http://localhost:4000/v4
62+
export TC_API_V5_BASE_URL=http://localhost:4000/v5
63+
```
64+
65+
- set M2M config params:
66+
```
67+
export AUTH0_CLIENT_ID=dummy
68+
export AUTH0_CLIENT_SECRET=dummy
69+
export AUTH0_URL=dummy
70+
export AUTH0_AUDIENCE=dummy
71+
```
72+
73+
- install dependencies `npm i`
74+
- run code lint check `npm run lint`
75+
- fix some lint errors `npm run lint:fix`
76+
- create db tables if not present `node test/init-db`, this is needed only for local test, in production the tables are already present
77+
- start notification consumer `npm run startConsumer`
78+
79+
80+
## Verification
81+
82+
- Run Kafka console producer to write message to topic `notifications.community.challenge.created`:
83+
84+
```
85+
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic notifications.community.challenge.created
86+
```
87+
88+
- Write message of challenge created:
89+
90+
```
91+
{ "topic": "notifications.community.challenge.created", "originator": "tc-direct", "timestamp": "2018-02-16T00:00:00", "mime-type": "application/json", "payload": { "challengeId": 30054674, "challengeTitle": "test", "challengeUrl": "http://www.topcoder.com/123", "userId": 8547899, "initiatorUserId": 123, "skills": ["dotnet", "xcode"] } }
92+
```
93+
94+
- You will see logging in the app console:
95+
96+
```
97+
info: Run handler handleChallengeCreated
98+
...
99+
verbose: Searched users: ...
100+
...
101+
info: Successfully sent notifications.action.email.connect.project.notifications.generic event with body ... to bus api
102+
...
103+
error: Failed to send email to user id: 5, handle: handle5
104+
...
105+
info: Saved 8 notifications for users: 1, 2, 3, 4, 5, 6, 7, 8
106+
info: Handler handleChallengeCreated was run successfully
107+
```
108+
109+
110+
- Run Kafka console producer to write message to topic `notifications.community.challenge.phasewarning`:
111+
112+
```
113+
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic notifications.community.challenge.phasewarning
114+
```
115+
116+
- Write message of challenge phase warning:
117+
118+
```
119+
{ "topic": "notifications.community.challenge.phasewarning", "originator": "tc-autopilot", "timestamp": "2018-02-16T00:00:00", "mime-type": "application/json", "payload": { "challengeId": 30054674, "challengeTitle": "test", "challengeUrl": "http://www.topcoder.com/123", "phase": "Submission", "remainingTime": 12345, "userId": 8547899, "initiatorUserId": 123 } }
120+
```
121+
122+
- You will see logging in the app console:
123+
124+
```
125+
info: Run handler handleChallengePhaseWarning
126+
...
127+
verbose: Searched users: ...
128+
...
129+
info: Successfully sent notifications.action.email.connect.project.notifications.generic event with body ... to bus api
130+
...
131+
error: Failed to send email to user id: 5, handle: handle5
132+
...
133+
info: Saved 8 notifications for users: 1, 2, 3, 4, 5, 6, 7, 8
134+
info: Handler handleChallengePhaseWarning was run successfully
135+
```
136+
137+
138+
- Write message of challenge retrieved with error:
139+
140+
```
141+
{ "topic": "notifications.community.challenge.phasewarning", "originator": "tc-autopilot", "timestamp": "2018-02-16T00:00:00", "mime-type": "application/json", "payload": { "challengeId": 1111, "challengeTitle": "test", "challengeUrl": "http://www.topcoder.com/123", "phase": "Submission", "remainingTime": 12345, "userId": 8547899, "initiatorUserId": 123 } }
142+
```
143+
144+
- You will see logging in the app console:
145+
146+
```
147+
info: Run handler handleChallengePhaseWarning
148+
...
149+
error: Handler handleChallengePhaseWarning failed
150+
...
151+
error: { Error: Internal Server Error ...
152+
```
153+
154+
155+
- Write message of challenge which is not found:
156+
157+
```
158+
{ "topic": "notifications.community.challenge.phasewarning", "originator": "tc-autopilot", "timestamp": "2018-02-16T00:00:00", "mime-type": "application/json", "payload": { "challengeId": 2222, "challengeTitle": "test", "challengeUrl": "http://www.topcoder.com/123", "phase": "Submission", "remainingTime": 12345, "userId": 8547899, "initiatorUserId": 123 } }
159+
```
160+
161+
- You will see logging in the app console:
162+
163+
```
164+
info: Run handler handleChallengePhaseWarning
165+
...
166+
error: Handler handleChallengePhaseWarning failed
167+
...
168+
error: { Error: Not Found ...
169+
```
170+
171+
172+
- Write message of challenge of id 3333:
173+
174+
```
175+
{ "topic": "notifications.community.challenge.phasewarning", "originator": "tc-autopilot", "timestamp": "2018-02-16T00:00:00", "mime-type": "application/json", "payload": { "challengeId": 3333, "challengeTitle": "test", "challengeUrl": "http://www.topcoder.com/123", "phase": "Submission", "remainingTime": 12345, "userId": 8547899, "initiatorUserId": 123 } }
176+
```
177+
178+
- You will see logging in the app console:
179+
180+
```
181+
info: Run handler handleChallengePhaseWarning
182+
...
183+
error: Handler handleChallengePhaseWarning failed
184+
...
185+
error: { Error: Internal Server Error ...
186+
... { message: 'there is some error' } ...
187+
... Error: cannot GET /v3/members/_search?query=handle:%22handle1%22%20OR%20handle:%22handle2%22%20OR%20handle:%22handle3%22&offset=0&limit=5&fields=userId,email,handle,firstName,lastName,photoURL,status (500) ...
188+
...
189+
```
190+
191+
192+
- You may write some invalid messages like below:
193+
194+
```
195+
{ "topic": "notifications.community.challenge.phasewarning", "originator": "tc-autopilot", "timestamp": "invalid", "mime-type": "application/json", "payload": { "challengeId": 30054674, "challengeTitle": "test", "challengeUrl": "http://www.topcoder.com/123", "phase": "Submission", "remainingTime": 12345, "userId": 8547899, "initiatorUserId": 123 } }
196+
```
197+
198+
```
199+
{ "topic": "notifications.community.challenge.phasewarning", "originator": "tc-autopilot", "timestamp": "2018-02-16T00:00:00", "mime-type": "application/json", "payload": { "challengeTitle": "test", "challengeUrl": "http://www.topcoder.com/123", "phase": "Submission", "remainingTime": 12345, "userId": 8547899, "initiatorUserId": 123 } }
200+
```
201+
202+
```
203+
{ [ xyz
204+
```
205+
206+
- You will see error logging in the app console.
207+
208+
- Use some PostgreSQL client to connect to the database, e.g. you may use the PostgreSQL's built-in client psql to connect to the database: `psql -U postgres`
209+
210+
- connect to database: `\c notification`
211+
212+
- select notifications: `select * from "Notifications";`
213+
214+
- you will see notification records:
215+
216+
```
217+
1 | 23154497 | notifications.community.challenge.created | {"skills": ["dotnet", "xcode"], "userId": 8547899, "challengeId": 30054522, "challengeUrl": "http://www.topcoder.com/123", "challengeTitle": "test", "initiatorUserId": 123} | f | f | | 2019-04-01 19:49:08.232+08 | 2019-04-01 19:49:08.232+08
218+
2 | 294446 | notifications.community.challenge.created | {"skills": ["dotnet", "xcode"], "userId": 8547899, "challengeId": 30054522, "challengeUrl": "http://www.topcoder.com/123", "challengeTitle": "test", "initiatorUserId": 123} | f | f | | 2019-04-01 19:49:08.232+08 | 2019-04-01 19:49:08.232+08
219+
...
220+
```
221+

‎README.md

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ tc-notifications (as a standard nodejs app) provides generic framework around no
2020
5. Either add deployment for this new notification consumer/processor in existing deployment script (if you want to host the processor as separate service in the same ECS cluster) or write a new script if you want to keep the deployment separate.
2121

2222
## Dependencies
23-
- nodejs https://nodejs.org/en/ (v6+)
23+
- nodejs https://nodejs.org/en/ (v6+, if newer version of node is used, e.g. v10, then it needs to install extra lib `npm i [email protected]` to support the gulp build)
2424
- Heroku Toolbelt https://toolbelt.heroku.com
2525
- git
2626
- PostgreSQL 9.5
@@ -50,6 +50,8 @@ The following parameters can be set in config files or in env variables:
5050
if not provided, then SSL connection is not used, direct insecure connection is used;
5151
if provided, it can be either path to private key file or private key content
5252
- **Topcoder API**
53+
- `TC_API_V3_BASE_URL`: the TopCoder API V3 base URL
54+
- `TC_API_V4_BASE_URL`: the TopCoder API V4 base URL
5355
- `TC_API_V5_BASE_URL`: the TopCoder API V5 base URL
5456
- **Notifications API**
5557
- `API_CONTEXT_PATH`: path to serve API on
@@ -59,6 +61,16 @@ The following parameters can be set in config files or in env variables:
5961
- `TOKEN_CACHE_TIME`: time period of the cached token
6062
- `AUTH0_CLIENT_ID`: auth0 client id
6163
- `AUTH0_CLIENT_SECRET`: auth0 client secret
64+
- `AUTH0_PROXY_SERVER_URL`: auth0 proxy server URL
65+
- **Consumer handlers**
66+
- `KAFKA_CONSUMER_HANDLERS`: mapping from consumer topic to handlers
67+
- **Email notification**
68+
- `ENV`: used to construct email category
69+
- `ENABLE_EMAILS`: whether to enable email notifications
70+
- `ENABLE_DEV_MODE`: whether to enable dev mode
71+
- `DEV_MODE_EMAIL`: recipient email used in dev mode
72+
- `DEFAULT_REPLY_EMAIL`: default reply email
73+
6274

6375
### Connect notification server
6476
Configuration for the connect notification server is at `connect/config.js`.

‎config/default.js

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,9 @@ module.exports = {
2828
KAFKA_CLIENT_CERT_KEY: process.env.KAFKA_CLIENT_CERT_KEY ?
2929
process.env.KAFKA_CLIENT_CERT_KEY.replace('\\n', '\n') : null,
3030

31-
TC_API_V5_BASE_URL: process.env.TC_API_V5_BASE_URL || 'https://api.topcoder-dev.com/v5',
31+
TC_API_V3_BASE_URL: process.env.TC_API_V3_BASE_URL || '',
32+
TC_API_V4_BASE_URL: process.env.TC_API_V4_BASE_URL || '',
33+
TC_API_V5_BASE_URL: process.env.TC_API_V5_BASE_URL || '',
3234
API_CONTEXT_PATH: process.env.API_CONTEXT_PATH || '/v5/notifications',
3335

3436
// Configuration for generating machine to machine auth0 token.
@@ -40,5 +42,19 @@ module.exports = {
4042
TOKEN_CACHE_TIME: process.env.TOKEN_CACHE_TIME || 86400000,
4143
AUTH0_CLIENT_ID: process.env.AUTH0_CLIENT_ID,
4244
AUTH0_CLIENT_SECRET: process.env.AUTH0_CLIENT_SECRET,
43-
AUTH0_PROXY_SERVER_URL: process.env.AUTH0_PROXY_SERVER_URL || '',
45+
AUTH0_PROXY_SERVER_URL: process.env.AUTH0_PROXY_SERVER_URL,
46+
47+
KAFKA_CONSUMER_RULESETS: {
48+
// key is Kafka topic name, value is array of ruleset which have key as handler function name defined in src/processors/index.js
49+
'challenge.notification.events' : [{handleChallenge : {type:'UPDATE_DRAFT_CHALLENGE', roles: ["Submitter" /** Competitor */, "Copilot", "Reviewer"]}}],
50+
//'notifications.community.challenge.created': ['handleChallengeCreated'],
51+
//'notifications.community.challenge.phasewarning': ['handleChallengePhaseWarning'],
52+
},
53+
54+
// email notification service related variables
55+
ENV: process.env.ENV,
56+
ENABLE_EMAILS: process.env.ENABLE_EMAILS ? Boolean(process.env.ENABLE_EMAILS) : false,
57+
ENABLE_DEV_MODE: process.env.ENABLE_DEV_MODE ? Boolean(process.env.ENABLE_DEV_MODE) : true,
58+
DEV_MODE_EMAIL: process.env.DEV_MODE_EMAIL,
59+
DEFAULT_REPLY_EMAIL: process.env.DEFAULT_REPLY_EMAIL,
4460
};

‎constants.js

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
module.exports = {
2+
// set to a small value in order to test pagination functionalities, set to larger value in production
3+
SEARCH_USERS_PAGE_SIZE: 5,
4+
5+
SETTINGS_EMAIL_SERVICE_ID: 'email',
6+
ACTIVE_USER_STATUSES: ['ACTIVE'],
7+
8+
BUS_API_EVENT: {
9+
EMAIL: {
10+
GENERAL: 'notifications.action.email.connect.project.notifications.generic',
11+
},
12+
},
13+
};

0 commit comments

Comments
 (0)