3.1.5 Transform Node and Python
I am trying to use the new 3.1.5 version to use a python script to calculate the last x amount of days from today, I basically want a script to determine the last x amount of days via a python script and then filter out only those records. I need to use a script and not the filters, because it will vary from Last 7, Last 28, Last Year.
I am thinking it involves importing datetime and then subtracting a time delta, but the node seems to be more complicated now that python needs to be configured and I can't even figure out how to start with the script because it looks like it requires 3 different scripts.
Can someone at Lavastorm do a video example of how to work with the PYTHON nodes now? It looks like a multi step process to even get a script to work. Thank you.
-
Hi Nancy,
Your approach is correct. I just tried the following and it worked for me:
In ConfigureFields field of the Transform node I put the following:
---from datetime import datetime, timedelta
N = 2
date_N_days_ago = datetime.now() - timedelta(days=N)
---
I can then use the date_N_days_ago value in scripts in ProcessRecords and ConfigureFields. For example to output the date_N_days_ago value as a field I could do the following:
ConfigureFields:
----from datetime import datetime, timedelta
#define a new field of type datetime on output out1
out1.Date_N_Days_Ago = datetimeN = 2
date_N_days_ago = datetime.now() - timedelta(days=N)
---
ProcessRecords
---
#output the value of date_N_days_ago in field out1.Date_N_Days_Agoout1.Date_N_Days_Ago = date_N_days_ago
---
Let me know if this helps.
Mark -
Nancy,
If you want to suppress records in a transform node you need to set the output to None based on the condition you have identified. The example below based on Mark's above should give you all the information you need.
Step 1
Create a Create Data node with the following content in the Data field
id:int,create:datetime
1,2017-08-01 12:35:41
2,2017-09-01 02:00:00
3,2017-07-01 05:23:21
4,2017-04-17 18:12:21
5,2017-06-21 15:12:21Step 2
Add a Transform node with the following in ConfigureFields
from datetime import datetime, timedelta
out1 += in1
N = 100
date_N_days_ago = datetime.now() - timedelta(days=N)
And the following in ProcessRecords
out1 += in1
if in1.create < date_N_days_ago:
out1 = NoneNote the result will only output those records newer than 100 days.
Regards, Tim.
-
Hello, thanks for the tips.
This example works as reproduced but it's not working with my data. I have a datetime field but I am trying to filter off a converted date to string. So that's more than likely my problem,
So ideally I would want my filter to find the x number of days then filter off the new truncated date field I created.
When I output the datetime though I don't want to see the entire datetime format, just a truncated version of it.
-
Hi Nancy,
The data flow attached to this knowledgebase article provides some examples of extracting elements of a date object as a string. The example data in the article used a date object but the same code can be used to extract the corresponding elements from a datetime object.
There are a number of articles describing the use of Python within the Transform node in the Tips & Tricks section of the knowledge base.
Regards,
Adrian
-
Hi Nancy,
If you can provide an example or two of what the data looks like in the string, that will help.
Can you also clarify what you mean by these statements (with an example):
So ideally I would want my filter to find the x number of days then filter off the new truncated date field I created.
- What does the new truncated date field look like?
When I output the datetime though I don't want to see the entire datetime format, just a truncated version of it.- So you want it to look like the new truncated date field you mentioned in your previous statement?
Hopefully this will give us enough information to be able to help you accomplish what you want.
Thanks,
Catharine
Please sign in to leave a comment.
Comments
6 comments