SharePoint search result display as Model Dialog or in Pop Up window

You can achieve this by editing the XSLT in Search Core Results web part.
Follow below steps,

1.      Open search site and search for some word.

2.      Once you see results on your search page.

3.      Edit the page

4.      Edit the Search Core Results webpart

5.      Expand the Display Properties section

6.      Uncheck "Use Local Visualization"

7.      Click on XSL Editor.

From here, we need to modify the XSL, so that all search results will be opened in a modal dialog.
For this, now we will edit Result template ().

• This template starts off with a statement.
• Just after the block, REMOVE  the link tag  

  <a id="{concat($currentId,'_Title')}">

    <xsl:attribute name="href">

      <xsl:value-of  select="$url"/>

    </xsl:attribute>

    <xsl:attribute name="title">

      <xsl:value-of select="title"/>

    </xsl:attribute>

    <xsl:choose>

      <xsl:when test="hithighlightedproperties/HHTitle[. != '']">

        <xsl:call-template name="HitHighlighting">

          <xsl:with-param name="hh" select="hithighlightedproperties/HHTitle" />

        </xsl:call-template>

      </xsl:when>

      <xsl:otherwise>

        <xsl:value-of select="title"/>

      </xsl:otherwise>

    </xsl:choose>

  </a>

• And REPLACE it with the following markup

  <a id="{concat($currentId,'_Title')}" href="javascript:SP.UI.ModalDialog.ShowPopupDialog('{$url}')">

    <xsl:attribute name="title">

      <xsl:value-of select="title"/>

    </xsl:attribute>

    <xsl:choose>

      <xsl:when test="hithighlightedproperties/HHTitle[. != '']">

        <xsl:call-template name="HitHighlighting">

          <xsl:with-param name="hh" select="hithighlightedproperties/HHTitle" />

        </xsl:call-template>

      </xsl:when>

      <xsl:otherwise>

        <xsl:value-of select="title"/>

      </xsl:otherwise>

    </xsl:choose>

  </a>

• Close the XSL Editor and save the webparts properties
• Refresh the page to test it.