{"id":182,"date":"2015-03-22T15:02:38","date_gmt":"2015-03-22T14:02:38","guid":{"rendered":"http:\/\/graal.ens-lyon.fr\/~diet\/?page_id=182"},"modified":"2015-03-22T15:18:42","modified_gmt":"2015-03-22T14:18:42","slug":"dagda","status":"publish","type":"page","link":"https:\/\/graal.ens-lyon.fr\/diet\/?page_id=182","title":{"rendered":"DAGDA"},"content":{"rendered":"<div id=\"parent-fieldname-description\" class=\"documentDescription\">\n<table id=\"tablepress-C_DAGDA\" class=\"tablepress tablepress-id-C_DAGDA\">\n<thead>\n<tr class=\"row-1\">\n\t<th colspan=\"3\" class=\"column-1\">Partners \/ Contributors<\/th>\n<\/tr>\n<\/thead>\n<tbody class=\"row-striping row-hover\">\n<tr class=\"row-2\">\n\t<td class=\"column-1\">Ga\u00ebl Le Mahec<\/td><td class=\"column-2\">MIS (Universit\u00e9 de Picardie Jules Verne)<\/td><td class=\"column-3\"><a href=\"http:\/\/graal.ens-lyon.fr\/~diet\/wp-content\/uploads\/2015\/03\/logo_mis.png\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/graal.ens-lyon.fr\/~diet\/wp-content\/uploads\/2015\/03\/logo_mis.png\" alt=\"logo_mis\" width=\"171\" height=\"40\" class=\"aligncenter size-full wp-image-125\" \/><\/a><\/td>\n<\/tr>\n<tr class=\"row-3\">\n\t<td class=\"column-1\">Vincent Breton<\/td><td class=\"column-2\">IN2P3<\/td><td class=\"column-3\"><a href=\"http:\/\/graal.ens-lyon.fr\/~diet\/wp-content\/uploads\/2015\/03\/logo_in2p3.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/graal.ens-lyon.fr\/~diet\/wp-content\/uploads\/2015\/03\/logo_in2p3.jpg\" alt=\"Logo IN2P3\" width=\"181\" height=\"69\" class=\"aligncenter size-full wp-image-121\" \/><\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<!-- #tablepress-C_DAGDA from cache --><\/div>\n<div class=\"documentDescription\"><\/div>\n<h1 class=\"documentDescription\" style=\"text-align: left;\">DAGDA: Data Arrangement for Grid and Distributed Application<\/h1>\n<div id=\"content-core\">\n<div id=\"parent-fieldname-text\" class=\"plain\" style=\"text-align: justify;\">\n<p>DAGDA is now the data manager by default for the DIET middleware which allows data explicit or implicit replications and advanced data management on the grid. It was designed to be backward compatible with previously developed applications for DIET which benefit transparently of the data replications. DAGDA introduces some new data management features in DIET:<\/p>\n<ul>\n<li>Explicit or implicit data replications.<\/li>\n<li>A data status backup\/restoration system.<\/li>\n<li>File sharing between the nodes which can access to the same disk partition.<\/li>\n<li>Choice of a data replacement algorithm.<\/li>\n<li>High level configuration about the memory and disk space DIET should use for the data storage and transfers.<\/li>\n<\/ul>\n<h2>DAGDA transfer model<\/h2>\n<div id=\"attachment_13\" style=\"width: 755px\" class=\"wp-caption aligncenter\"><a href=\"http:\/\/graal.ens-lyon.fr\/~diet\/wp-content\/uploads\/2015\/03\/dagda-archi.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-13\" class=\"wp-image-13 size-full\" src=\"http:\/\/graal.ens-lyon.fr\/~diet\/wp-content\/uploads\/2015\/03\/dagda-archi.png\" alt=\"DAGDA Architecture\" width=\"745\" height=\"646\" \/><\/a><p id=\"caption-attachment-13\" class=\"wp-caption-text\">Figure 1: DAGDA Architecture<\/p><\/div>\n<p>&nbsp;<\/p>\n<p>To transfer a data, DAGDA uses the <tt>pull model<\/tt> instead of the <tt>push model<\/tt> used by DTM: The data are not sent into the profile from the source to the destination, but they are downloaded by the destination from the source. Figure 1 presents how DAGDA manages the data transfers for a standard DIET call:<\/p>\n<ol type=\"1\">\n<li>The client performs a DIET service call.<\/li>\n<li>DIET selects one or more SeDs to execute the service.<\/li>\n<li>The client submits its request to the selected SeD, sending only the data descriptions.<\/li>\n<li>The SeD downloads the new data from the client and the persistent ones from the nodes on which they are stored.<\/li>\n<li>The SeD executes the service using the input data.<\/li>\n<li>The SeD performs updates on the inout and out data and sends their descriptions to the client. The client then downloads the <var>volatile<\/var> and <var>persistent return<\/var> data.<\/li>\n<\/ol>\n<p>At each step of the submission, the transfers are allways launched by the destination node. All the transfers operations are transparently performed and the user does not have to modify its application which uses data persistency to take benefits of the data replications.<\/p>\n<p><b>Remark<\/b>: <var>After the call, the persistent output data obtained from other nodes are replicated on the SeD and will stay on it until they are erased by the user or by the data replacement algorithm.<\/var><\/p>\n<div class=\"section\">\n<h3>DAGDA architecture<\/h3>\n<\/div>\n<div class=\"section\">Each DAGDA data manager uses the same implementation presented on the Figure 2.<\/div>\n<div class=\"section\"><\/div>\n<div class=\"section\">\n<div id=\"attachment_16\" style=\"width: 537px\" class=\"wp-caption aligncenter\"><a href=\"http:\/\/graal.ens-lyon.fr\/~diet\/wp-content\/uploads\/2015\/03\/dagda-general.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-16\" class=\"size-full wp-image-16\" src=\"http:\/\/graal.ens-lyon.fr\/~diet\/wp-content\/uploads\/2015\/03\/dagda-general.png\" alt=\"DAGDA Overview \" width=\"527\" height=\"445\" \/><\/a><p id=\"caption-attachment-16\" class=\"wp-caption-text\">Figure 2: DAGDA Overview<\/p><\/div>\n<p>&nbsp;<\/p>\n<ul>\n<li>The DAGDA object associates an ID to each stored data, manages the transfers by choosing the \u00ab\u00a0best\u00a0\u00bb data source according to statistics about the previous transfers time and performs the data searches.<\/li>\n<li>The CORBA interface is used for inter-nodes communications.<\/li>\n<li>The storage resources management ensures that the data never exceed the size allowed by the user and replace a data when necessary.<\/li>\n<li>The DAGDA API allows the user to perform some data management tasks:\n<ul>\n<li>To add a data into the DAGDA hierarchy.<\/li>\n<li>To get a data from the DAGDA hierarchy.<\/li>\n<li>To replicate a data on chosen nodes.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>Currently DAGDA offers three data replacement algorithms: <b>L<\/b>east <b>R<\/b>ecently <b>U<\/b>sed (LRU), <b>L<\/b>east <b>F<\/b>requently <b>U<\/b>sed (LFU) and <b>F<\/b>irst <b>I<\/b>n <b>F<\/b>irst <b>O<\/b>ut (FIFO) :<\/p>\n<ul>\n<li>LRU: The least recently used data is removed to free the needed space for the new one.<\/li>\n<li>LFU: The least frequently used data is removed to free the needed space.<\/li>\n<li>FIFO: The <var>\u00ab\u00a0oldest\u00a0\u00bb<\/var> data is removed to free the needed space.<\/li>\n<\/ul>\n<p>These algorithms select the data to be removed only among the <var>non-sticky<\/var> ones of sufficient size to store the new data.<\/p>\n<p>&nbsp;<\/p>\n<h3>Interface with DIET<\/h3>\n<p>DAGDA is designed to be backward compatible with the previous DIET client-server applications. Moreover, DAGDA extends the standard DIET API with new data management functions. A DAGDA object is interfaced with DIET at each level of the DIET hierarchy.<\/p>\n<div class=\"section\">\n<h4>DAGDA interactions with a client application<\/h4>\n<ul>\n<li>Without performing any change on a previous application that was running with DTM. When the service is called, DIET adds the user data in the client DAGDA component and sends the profile description only.<\/li>\n<li>Using the API, no service call is needed to manage the data. The user can add a data, replicate a persistent data already in the DAGDA hierarchy or get a data using direct calls to DAGDA.<\/li>\n<\/ul>\n<div id=\"attachment_14\" style=\"width: 416px\" class=\"wp-caption aligncenter\"><a href=\"http:\/\/graal.ens-lyon.fr\/~diet\/wp-content\/uploads\/2015\/03\/dagda-client.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-14\" class=\"size-full wp-image-14\" src=\"http:\/\/graal.ens-lyon.fr\/~diet\/wp-content\/uploads\/2015\/03\/dagda-client.png\" alt=\"Client Dagda\" width=\"406\" height=\"199\" \/><\/a><p id=\"caption-attachment-14\" class=\"wp-caption-text\">Figure 3: DAGDA Client<\/p><\/div>\n<div class=\"section\">\n<h4>DAGDA interactions with a server application.<\/h4>\n<ul>\n<li>Using a simple service call, DIET gets the data from the server DAGDA component. Then DAGDA transfers the data from the client or from a selected source among the DAGDA nodes. The profile is then filled with references to the data and the server developer can use them transparently.<\/li>\n<li>Using the API, the user can add a data, replicate a persistent data already in the DAGDA hierarchy or get a data using direct calls to DAGDA.<\/li>\n<\/ul>\n<div id=\"attachment_17\" style=\"width: 416px\" class=\"wp-caption aligncenter\"><a href=\"http:\/\/graal.ens-lyon.fr\/~diet\/wp-content\/uploads\/2015\/03\/dagda-sed.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-17\" class=\"size-full wp-image-17\" src=\"http:\/\/graal.ens-lyon.fr\/~diet\/wp-content\/uploads\/2015\/03\/dagda-sed.png\" alt=\"DAGDA SeD\" width=\"406\" height=\"199\" \/><\/a><p id=\"caption-attachment-17\" class=\"wp-caption-text\">Figure 4: DAGDA SeD<\/p><\/div>\n<h4>DAGDA interactions with a DIET agent.<\/h4>\n<ul>\n<li>As in the other elements of the DIET hierarchy, the scheduler developer can add a data, replicate a persistent data already in the DAGDA hierarchy or get a data using direct calls to DAGDA.<\/li>\n<\/ul>\n<p>All the data management function of the API can be used synchronously or asynchronously. For the server and scheduler sides, the tranfers can also be done asynchronously without waiting for their ends.<\/p>\n<div id=\"attachment_12\" style=\"width: 384px\" class=\"wp-caption aligncenter\"><a href=\"http:\/\/graal.ens-lyon.fr\/~diet\/wp-content\/uploads\/2015\/03\/dagda-agent.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-12\" class=\"size-full wp-image-12\" src=\"http:\/\/graal.ens-lyon.fr\/~diet\/wp-content\/uploads\/2015\/03\/dagda-agent.png\" alt=\"Agent DAGDA\" width=\"374\" height=\"126\" \/><\/a><p id=\"caption-attachment-12\" class=\"wp-caption-text\">Figure 5: Agent DAGDA<\/p><\/div>\n<div class=\"section\">\n<h3>Data Replication<\/h3>\n<p>DAGDA allows the users to replicate the data stored on the platform for performance improvements. To select the nodes where the data should be replicated, the replication function takes a replication rule as a parameter. This string is constructed as follows:<\/p>\n<ul>\n<li>The node identifier to use for the node selection: <b>ID<\/b> or <b>hostname<\/b>.<\/li>\n<li>A string that can use wildcard characters for a <var>glob-like<\/var> pattern matching.<\/li>\n<li>A keyword to choose whether DAGDA should remove a data using the data replacement algorithm if necessary: <b>replace<\/b> or <b>noreplace<\/b>.<\/li>\n<\/ul>\n<p>These three parameters have to be separated by the \u00ab\u00a0:\u00a0\u00bb delimiter character. For example, <var>\u00ab\u00a0hostname:*-??.lyon.grid5000.fr:noreplace\u00a0\u00bb<\/var> is a valid replication rule string which means that the data should be replicated on all the nodes having a <var>hostname<\/var> matching <var>*-??.lyon.grid5000.fr<\/var> without removing a data if the free space is not sufficient.<\/p>\n<\/div>\n<div class=\"section\">\n<h3>File sharing<\/h3>\n<div id=\"attachment_15\" style=\"width: 557px\" class=\"wp-caption aligncenter\"><a href=\"http:\/\/graal.ens-lyon.fr\/~diet\/wp-content\/uploads\/2015\/03\/dagda-file_sharing.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-15\" class=\"size-full wp-image-15\" src=\"http:\/\/graal.ens-lyon.fr\/~diet\/wp-content\/uploads\/2015\/03\/dagda-file_sharing.png\" alt=\"DAGDA file sharing\" width=\"547\" height=\"302\" \/><\/a><p id=\"caption-attachment-15\" class=\"wp-caption-text\">Figure 6: DAGDA file sharing<\/p><\/div>\n<div class=\"section\">\n<p>It is frequent that several nodes can access a shared disk partition (through a NFS server on a cluster for example). With DAGDA, a node can be configured to share the files that it manages with all its children nodes. A typical example of the usage of these features is for a service using a large file as read-only parameter executed on several SeD located on the same cluster. Figure 6 presents such a file sharing between several nodes:<\/p>\n<ul>\n<li>Two Local Agents and six SeDs can access to the same NFS partition (blue lines).<\/li>\n<li>The files registered on the first LA (red lines) are declared as shared with all of its children.<\/li>\n<li>If one of the SeDs or the second LA has to access to one of the files registered on the first LA, they can access it directly without data transfer through DIET.<\/li>\n<\/ul>\n<\/div>\n<div class=\"section\">\n<h3>Status backup\/restoration<\/h3>\n<p>The user can configure the SeDs and Agents to store their data on a selected file. Then the client can command to all the nodes that allow it to save their current data status. These nodes can then be stopped and restarted restoring their data status. This command allows the user to stop temporary DIET preserving the data status of the chosen nodes after restart.<\/p>\n<\/div>\n<div class=\"section\">\n<h3>Physical resources usage configuration<\/h3>\n<p>The users can fix several parameters about the resources DAGDA is allowed to use:<\/p>\n<ul>\n<li>The maximum disk space DAGDA can use to store files.<\/li>\n<li>The maximum memory space DAGDA can use to store data on memory.<\/li>\n<li>The maximum CORBA message size DAGDA can send on the network.<\/li>\n<\/ul>\n<p>All these parameters are optionnal. If the maximum CORBA message size exceeds the maximum GIOP size allowed by the ORB, DAGDA uses this last value avoiding CORBA errors that could occur when using DTM. This default behaviour avoids the users to modify the ORB configuration. DAGDA performs the data transfers in several parts if necessary, so sending a data larger than the maximum CORBA message does not cause any error. Moreover, the measured overhead caused by this behaviour is negligible if the maximum message size is larger than some kilobytes.<\/p>\n<h2>Contact<\/h2>\n<div class=\"section\">\n<p>Questions about DAGDA can be directed to <a href=\"&#x6d;&#x61;&#x69;&#108;&#116;&#111;:d&#x69;&#x65;&#x74;&#x2d;&#117;&#115;r&#64;&#x6c;&#x69;&#x73;&#x74;&#101;&#46;en&#x73;&#x2d;&#x6c;&#x79;&#111;&#110;&#46;fr\">die&#116;&#45;&#117;&#115;&#x72;&#x40;&#x6c;&#x69;&#x73;&#x74;&#x65;&#46;en&#115;&#45;&#108;&#121;&#111;&#x6e;&#x2e;&#x66;&#x72;<\/a> or to\u00a0 <span class=\"link-mailto\"><a class=\"externalLink\" href=\"m&#97;&#x69;&#x6c;&#x74;o&#58;&#71;&#x61;&#x65;l&#46;&#76;&#x65;&#x6d;ah&#101;&#x63;&#x40;&#x75;-&#112;&#x69;&#x63;&#x61;r&#100;&#105;&#x65;&#x2e;f&#114;\">Gael Le Mahec<\/a><\/span><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>DAGDA: Data Arrangement for Grid and Distributed Application DAGDA is now the data manager by default for the DIET middleware which allows data explicit or implicit replications and advanced data management on the grid. It was designed to be backward compatible with previously developed applications for DIET which benefit transparently of the data replications. DAGDA [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":178,"menu_order":1,"comment_status":"closed","ping_status":"open","template":"","meta":{"footnotes":""},"class_list":["post-182","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/graal.ens-lyon.fr\/diet\/index.php?rest_route=\/wp\/v2\/pages\/182","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/graal.ens-lyon.fr\/diet\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/graal.ens-lyon.fr\/diet\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/graal.ens-lyon.fr\/diet\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/graal.ens-lyon.fr\/diet\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=182"}],"version-history":[{"count":8,"href":"https:\/\/graal.ens-lyon.fr\/diet\/index.php?rest_route=\/wp\/v2\/pages\/182\/revisions"}],"predecessor-version":[{"id":205,"href":"https:\/\/graal.ens-lyon.fr\/diet\/index.php?rest_route=\/wp\/v2\/pages\/182\/revisions\/205"}],"up":[{"embeddable":true,"href":"https:\/\/graal.ens-lyon.fr\/diet\/index.php?rest_route=\/wp\/v2\/pages\/178"}],"wp:attachment":[{"href":"https:\/\/graal.ens-lyon.fr\/diet\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=182"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}