Display Lists and Libraries in sharepoint search refinements.

You can achieve this by editing the XSLT in Search Core Results web part.
Follow below steps,

• Open search site and search for some word.
• Once you see results on your search page.
• Edit the page.
• Now edit Refinement Panel -> Edit Web Part.
• Expand Refinement section.
• Copy XML from Filter Category Definition to XML editor.
• Add new category under <FilterCategories> section.
• Copy and Paste below section.

<Category    Title="Site"    Description="Which site this document is from" 

    Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator"     

MetadataThreshold="5"    NumberOfFiltersToDisplay="4"    MaxNumberOfFilters="20"

     SortBy="Frequency"    SortByForMoreFilters="Name"    SortDirection="Descending" 

    SortDirectionForMoreFilters="Ascending"    ShowMoreLink="True"    MappedProperty="SiteName"

     MoreLinkText="show more"    LessLinkText="show fewer" />

• Copy new XML and update Web Part’s Filter Category Definition.
• Close the XSL Editor and save the webparts properties.
• Refresh the page to test it.

SharePoint search result display as Model Dialog or in Pop Up window

You can achieve this by editing the XSLT in Search Core Results web part.
Follow below steps,

1.      Open search site and search for some word.

2.      Once you see results on your search page.

3.      Edit the page

4.      Edit the Search Core Results webpart

5.      Expand the Display Properties section

6.      Uncheck "Use Local Visualization"

7.      Click on XSL Editor.

From here, we need to modify the XSL, so that all search results will be opened in a modal dialog.
For this, now we will edit Result template ().

• This template starts off with a statement.
• Just after the block, REMOVE  the link tag  

  <a id="{concat($currentId,'_Title')}">

    <xsl:attribute name="href">

      <xsl:value-of  select="$url"/>

    </xsl:attribute>

    <xsl:attribute name="title">

      <xsl:value-of select="title"/>

    </xsl:attribute>

    <xsl:choose>

      <xsl:when test="hithighlightedproperties/HHTitle[. != '']">

        <xsl:call-template name="HitHighlighting">

          <xsl:with-param name="hh" select="hithighlightedproperties/HHTitle" />

        </xsl:call-template>

      </xsl:when>

      <xsl:otherwise>

        <xsl:value-of select="title"/>

      </xsl:otherwise>

    </xsl:choose>

  </a>

• And REPLACE it with the following markup

  <a id="{concat($currentId,'_Title')}" href="javascript:SP.UI.ModalDialog.ShowPopupDialog('{$url}')">

    <xsl:attribute name="title">

      <xsl:value-of select="title"/>

    </xsl:attribute>

    <xsl:choose>

      <xsl:when test="hithighlightedproperties/HHTitle[. != '']">

        <xsl:call-template name="HitHighlighting">

          <xsl:with-param name="hh" select="hithighlightedproperties/HHTitle" />

        </xsl:call-template>

      </xsl:when>

      <xsl:otherwise>

        <xsl:value-of select="title"/>

      </xsl:otherwise>

    </xsl:choose>

  </a>

• Close the XSL Editor and save the webparts properties
• Refresh the page to test it.