How do I communicate a page(modal dialog) with its sibling (sidebar)?

Question 1

Perhaps it's not really its parent and that's the reason for it not working.

Right - there isn't actually a DOM parent / child relationship in play here. Both the sidebar and the modal dialog were launched from server-side scripts, and are independent. They can both communicate with the server, though, so you can use a store-and-forward technique to get the result from your picker to the sidebar.

Basic idea:

The sidebar will start polling the server for the picker's result as soon as it requests the picker's launch.
The picker's result will be sent to the server, using google.script.run.
The server will store the result temporarily - this could be as simple as a global variable, depending on your situation.
Once there is a result, the next poll will retrieve it.

Have a look at How to poll a Google Doc from an add-on for the basic idea of a poller.

Message Sequence Chart

Question 2

Issue:

Sidebar and modal dialog(siblings) are not able to communicate despite having same origin.

Solution:

It is possible to get a reference to the sidebar html from modal dialog through the ancestor parent window.top, even though the parent is cross origin. From there, it is possible to

directly communicate with each other
use window.postMessage() to communicate with each other

Without a reference to each other, it is still possible to communicate with each other through

the server and script properties service. However, Here, one of them needs to poll the server at set intervals to get any updates from the other(as illustrated here).
use cookies/localstorage to communicate with each other

To Read:

Sample script(using direct access through cross origin frame `window.top`):

addOn.html[Sidebar]

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width" />
    <title>Addon</title>
    <style>
      #spinner {
        display: none;
        background-color: tomato;
        position: absolute;
        top: 1%;
        width: 100%;
        justify-items: center;
      }
    </style>
  </head>
  <body>
    <div id="spinner"><p>Loading modal dialog...</p></div>
    <div id="output"></div>
    <script charset="utf-8">
      google.script.run.withSuccessHandler(spinner).testModal();
      function spinner(e) {
        document.getElementById('spinner').style.display = e || 'flex';
      }
      (async () => {
        //After modal dialog has finished, receiver will be resolved
        let receiver = new Promise((res, rej) => {
          window.modalDone = res;
        });
        var message = await receiver;
        document.querySelector('#output').innerHTML = message;
        //Do what you want here
      })();
    </script>
  </body>
</html>

modalAddOn.html[Modal dialog/picker]

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <title></title>
  </head>
  <body>
    Modal Dialog
    <script>
      (function findSideBar(limit) {
        let f = window.top.frames;
        for (let i = 0; i < limit; ++i) {
          try {
            if (
              f[i] /*/iframedAppPanel*/ &&
              f[i].length &&
              f[i][0] && //#sandboxFrame
              f[i][0][0] && //#userHtmlFrame
              window !== f[i][0][0] //!== self
            ) {
              console.info('Sidebar found ');
              alert('Removing loadbar and closing self');
              var sidebar = f[i][0][0];
              sidebar.spinner('none'); //Remove sidebar spinner
              sidebar.modalDone('Modal says Hi'); //Modal has finished
              google.script.host.close();
            }
          } catch (e) {
            console.error(e);
            continue;
          }
        }
      })(10);
    </script>
  </body>
</html>

code.gs[Server]

function testModal() {
  SpreadsheetApp.getUi().showModelessDialog(
    HtmlService.createHtmlOutputFromFile('modalAddOn')
      .setHeight(500)
      .setWidth(300),
    ' '
  );
}

function onOpen(e) {
  SpreadsheetApp.getUi()
    .createMenu('Sidebar')
    .addItem('Show Add-On', 'showSidebar')
    .addToUi();
}

function showSidebar() {
  SpreadsheetApp.getUi().showSidebar(
    HtmlService.createTemplateFromFile('addOn.html').evaluate()
  );
}

How do I communicate a page(modal dialog) with its sibling (sidebar)?

Issue:

Solution:

To Read:

Sample script(using direct access through cross origin frame window.top):

Related questions:

Sample script(using direct access through cross origin frame `window.top`):