![]() ![]() This is mostly taken out of the yaml file that I was using. Quick first logical step would be to modify the airflow deployment yaml file to include these things when creating the image. I have to convert it to int, but other than that, I just want to load it. I just load the env var from PONY_RUN_AS_USER and just use it. So, if you see, things are relatively simple. Here is a small example of where I use it. You can also use this to template variables that I have mentioned earlier using var. The config will be the json object and you can drill down any levels deep with just the. ![]() You can use this to template out things from the config that gets passed in when you trigger a job from the UI or using the API. You can also use this for two other things. Things like run_id or ds(datetime stamp). I use this primary for values that are specific to the runtime of a airflow job. You can read more about it in the Airflow reference for Jinja Templating and macros. It is kinda restricted, but it does provide some nice conveniences. get ( "pony_builder_image_name" ), # accessing variableĪirflow actually uses Jinja templates for some stuff. Once set, you can use an airflow variable like below: with DAG ( There you can create edit and delete and env var. In the UI, it is available under Admin>Variables. It is also possible to set them via env variables if we prefix the env var with AIRFLOW_VAR_. These are values that can be set from the Airflow UI. I mainly use this for image_pull_secrets and namespace as they were same as the airflow webserver in my case. This works great if we just want to reuse things from the airflow config. If you see the example above, you can see how the image_pull_secrets value is fetched form the config. ![]() get ( "kubernetes", "image_pull_secrets" ) , configuration import confĭag_id = "my-little-pony", schedule_interval = None, start_date =YESTERDAY This can be done with something like below: from airflow. One easy place where we can pull a config from is the airflow.cfg file. What I want to go over in this blog is ways in which we can get different variables into a dag which has KubernetesPodOperator. I am not saying this is something as important as that, but it took me a while to figure it out and I thought I should document it somewhere. If you try really hard, you will be able to find even a way to capture Mewtwo. There is always some way to do everything. With all that said, you can do everything that you want to do in Airflow. I don't really like it to be frank, but I am stuck with it for now. I have been working with Airflow for quite a while. The random_value parameter is set to the random macro, which generates a unique UUID.Templating things in Airflow DAG meain/blog The params argument in the DummyOperator allows us to pass in parameters that can be used in the task. In this example, the random macro is used to generate a unique UUID for each instance of task1. In this example, macros.uuid is passed as an argument to the log_uuid function. Here's an example of how you can use it in a PythonOperator: from airflow import DAGįrom _operator import PythonOperator To use macros.uuid in your Airflow tasks, you can reference it within your task's parameters or in a Jinja template. This can be useful in scenarios where you need to generate a unique identifier for each task instance, for example, when naming temporary files or tables. In Apache Airflow, macros.uuid is a built-in macro that generates a unique identifier (UUID) each time it's called. Using the macros.uuid Variable in Airflow The datetime.now() function is part of the datetime module, which is accessible through macros.time.įor more information on macros in Airflow, you can refer to the official Airflow documentation. In this example, the print_current_time function prints the current date and time when the task is executed. Here's an example of how you might use macros.time in a PythonOperator: from _operator import PythonOperatorĭag = DAG('time_dag', start_date=datetime(2022, 1, 1)) For example, you might need to generate a timestamp, calculate a time difference, or format a date string. This variable is particularly useful when you need to perform operations related to date and time within your tasks. It provides access to Python's built-in datetime module, which supplies classes for manipulating dates and times. ![]() The macros.time variable is one of the many predefined macros in Airflow. They are useful for dynamic programming and to ensure code reusability. In Apache Airflow, macros are a set of pre-defined variables and functions that can be used in your DAG definitions. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |